Nov 01 22:19:51.021225 ip-10-0-134-88 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 22:19:51.021235 ip-10-0-134-88 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 22:19:51.021243 ip-10-0-134-88 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 22:19:51.021536 ip-10-0-134-88 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 22:20:01.110686 ip-10-0-134-88 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 22:20:01.110707 ip-10-0-134-88 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5255c7cb0ed54d44b1cdbc10ea7d219b -- Nov 01 22:22:02.260247 ip-10-0-134-88 systemd[1]: Starting Kubernetes Kubelet... Nov 01 22:22:02.631977 ip-10-0-134-88 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:02.631977 ip-10-0-134-88 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 22:22:02.631977 ip-10-0-134-88 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:02.631977 ip-10-0-134-88 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:02.635663 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635570 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:02.635663 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635662 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:02.635663 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635666 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635669 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635672 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635675 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635678 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635681 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635684 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635687 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635690 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635693 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635695 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635698 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635701 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635704 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635707 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635709 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635712 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635714 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635717 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:02.635757 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635720 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635724 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635727 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635730 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635732 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635736 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635738 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635741 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635744 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635746 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635749 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635751 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635756 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635760 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635763 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635766 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635769 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635772 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635775 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635777 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:02.636242 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635780 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635782 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635785 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635787 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635790 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635793 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635795 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635798 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635800 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635803 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635806 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635808 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635812 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635815 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635818 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635821 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635824 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635826 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635829 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635831 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:02.636722 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635834 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635837 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635840 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635842 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635844 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635847 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635851 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635854 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635856 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635858 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635861 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635863 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635867 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635870 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635873 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635875 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635878 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635881 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635883 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:02.637253 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635886 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635888 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635891 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635894 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635910 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635914 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635916 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635919 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635922 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635924 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635927 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635929 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635932 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635935 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635937 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635940 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635943 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635947 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635951 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635954 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:02.637711 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635958 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635961 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635964 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635967 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635969 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.635973 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636798 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636807 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636810 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636814 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636817 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636820 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636823 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636826 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636828 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636831 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636835 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636838 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636841 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:02.638208 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636844 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636846 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636849 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636851 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636854 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636857 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636860 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636862 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636865 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636868 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636870 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636873 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636875 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636878 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636881 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636883 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636886 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636888 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636891 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:02.638671 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636909 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636913 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636917 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636920 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636923 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636925 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636928 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636931 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636933 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636936 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636939 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636942 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636946 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636949 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636952 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636954 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636957 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636960 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636962 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:02.639152 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636965 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636968 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636971 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636973 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636976 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636978 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636981 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636984 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636986 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636989 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636992 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636994 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.636997 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637000 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637002 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637005 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637007 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637010 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637013 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:02.639606 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637015 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637018 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637021 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637023 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637026 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637029 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637032 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637035 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637037 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637040 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637042 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637045 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637047 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637050 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637052 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637055 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637057 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637060 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637062 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637065 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:02.640074 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637067 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637070 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637073 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637075 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637078 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637081 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637084 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637087 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637089 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637092 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637094 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637097 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637101 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637105 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637108 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637111 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637653 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637660 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637665 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:02.640561 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637669 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637672 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637676 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637679 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637682 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637685 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637688 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637691 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637693 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637696 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637699 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637701 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637704 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637706 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637709 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637711 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637714 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637717 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637720 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:02.641039 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637723 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637726 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637729 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637732 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637734 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637737 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637739 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637742 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637745 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637748 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637750 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637760 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637763 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637766 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637769 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637771 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637774 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637777 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637779 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637782 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:02.641524 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637784 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637787 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637789 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637792 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637795 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637797 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637800 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637802 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637805 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637810 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637813 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637815 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637818 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637821 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637823 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637826 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637828 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637831 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637835 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637838 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:02.642018 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637840 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637843 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637846 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637848 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637850 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637853 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637856 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637859 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637861 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637864 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637867 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637869 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637872 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637874 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637877 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637879 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637881 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637884 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637887 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637889 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:02.642487 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637891 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637911 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637914 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637917 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637920 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637922 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637925 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637927 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637932 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637936 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637941 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637944 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637947 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637950 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637953 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637955 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637958 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637960 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637962 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:02.642999 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637966 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637968 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637971 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637973 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.637976 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639022 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639034 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639042 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639048 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639053 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639057 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639062 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639066 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639069 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639074 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639078 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639081 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639084 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639088 2580 flags.go:64] FLAG: --cgroup-root="" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639090 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639094 2580 flags.go:64] FLAG: --client-ca-file="" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639096 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639100 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 22:22:02.643452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639105 2580 flags.go:64] FLAG: --cluster-domain="" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639108 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639111 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639114 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639118 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639122 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639125 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639128 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639132 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639135 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639139 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639142 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639145 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639148 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639153 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639157 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639160 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639163 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639166 2580 flags.go:64] FLAG: --enable-server="true" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639170 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639174 2580 flags.go:64] FLAG: --event-burst="100" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639178 2580 flags.go:64] FLAG: --event-qps="50" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639181 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639184 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 22:22:02.644141 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639188 2580 flags.go:64] FLAG: --eviction-hard="" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639192 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639195 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639199 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639202 2580 flags.go:64] FLAG: --eviction-soft="" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639205 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639208 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639211 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639215 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639218 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639221 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639224 2580 flags.go:64] FLAG: --feature-gates="" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639228 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639231 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639235 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639238 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639241 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639245 2580 flags.go:64] FLAG: --help="false" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639248 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639252 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639255 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639259 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639262 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 22:22:02.644739 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639266 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639269 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639272 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639275 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639278 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639281 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639284 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639287 2580 flags.go:64] FLAG: --kube-reserved="" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639291 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639294 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639297 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639300 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639303 2580 flags.go:64] FLAG: --lock-file="" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639306 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639309 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639312 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639318 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639321 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639324 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639332 2580 flags.go:64] FLAG: --logging-format="text" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639335 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639338 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639341 2580 flags.go:64] FLAG: --manifest-url="" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639344 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639350 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 22:22:02.645311 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639353 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639357 2580 flags.go:64] FLAG: --max-pods="110" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639360 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639363 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639367 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639370 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639373 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639376 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639379 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639387 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639390 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639393 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639396 2580 flags.go:64] FLAG: --pod-cidr="" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639400 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639403 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639406 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639409 2580 flags.go:64] FLAG: --port="10250" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639412 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639415 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d733dfd78c2c1aca" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639418 2580 flags.go:64] FLAG: --qos-reserved="" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639421 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639424 2580 flags.go:64] FLAG: --register-node="true" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639427 2580 flags.go:64] FLAG: --register-with-taints="" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639431 2580 flags.go:64] FLAG: --registry-burst="10" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639434 2580 flags.go:64] FLAG: --registry-qps="5" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639437 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 01 22:22:02.645921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639440 2580 flags.go:64] FLAG: --reserved-memory="" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639444 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639447 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639450 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639453 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639456 2580 flags.go:64] FLAG: --runonce="false" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639460 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639465 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639468 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639471 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639474 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639478 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639481 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639485 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639487 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639491 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639494 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639497 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639500 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639503 2580 flags.go:64] FLAG: --system-cgroups="" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639506 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639512 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639515 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639518 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639521 2580 flags.go:64] FLAG: --tls-min-version="" Nov 01 22:22:02.646546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639524 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639528 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639531 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639535 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639539 2580 flags.go:64] FLAG: --v="2" Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639544 2580 flags.go:64] FLAG: --version="false" Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639548 2580 flags.go:64] FLAG: --vmodule="" Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639552 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.639556 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639697 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639700 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639703 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639706 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639709 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639713 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639716 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639719 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639721 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639724 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639727 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639730 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:02.647155 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639733 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639736 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639738 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639741 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639744 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639746 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639749 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639751 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639754 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639757 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639760 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639763 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639765 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639768 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639773 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639775 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639778 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639781 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639783 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639786 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:02.647699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639789 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639791 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639794 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639796 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639799 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639803 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639805 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639808 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639811 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639813 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639816 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639819 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639822 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639824 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639827 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639829 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639834 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639838 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639843 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:02.648209 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639846 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639849 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639851 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639854 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639857 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639859 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639862 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639866 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639868 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639871 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639874 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639876 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639879 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639881 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639884 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639887 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639889 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639892 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639908 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:02.648674 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639911 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639913 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639916 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639918 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639921 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639924 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639927 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639930 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639932 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639935 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639938 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639940 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639943 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639946 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639949 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639951 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639954 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639956 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639959 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639962 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:02.649140 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639966 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639969 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639972 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639974 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639977 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639979 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639982 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639985 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639988 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639990 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639993 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.639997 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640000 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640002 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640005 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640007 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:02.649610 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.640035 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640073 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640078 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640082 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640084 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640088 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640090 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640093 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640096 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640098 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640101 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640104 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640106 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640109 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640111 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640114 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640118 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640121 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640123 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640126 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:02.650012 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640128 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640131 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640134 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640136 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640138 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640141 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640144 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640146 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640150 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640153 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640155 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640158 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640161 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640163 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640166 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640168 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640172 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640174 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640177 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640179 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:02.650494 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640182 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640185 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640187 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640189 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640192 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640194 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640197 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640200 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640204 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640206 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640208 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640211 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640215 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640219 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640223 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640227 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640229 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640233 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640235 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:02.651046 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640238 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640242 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640244 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640247 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640250 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640252 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640255 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640257 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640260 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640263 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640266 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640269 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640271 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640274 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640276 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640279 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640281 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640285 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640288 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640291 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:02.651530 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640293 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640297 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640300 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640302 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640305 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640307 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640310 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640312 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640315 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640318 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640320 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640323 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640325 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640329 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640332 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640335 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640337 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640340 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640343 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640345 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:02.652317 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640348 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640350 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640353 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640356 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640359 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640362 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640366 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.640368 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.647851 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.647873 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.648621 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.648630 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648695 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648701 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648706 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648710 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648713 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648717 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648720 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:02.652877 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648722 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648725 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648728 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648731 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648734 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648736 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648739 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648742 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648745 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648747 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648750 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648753 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648756 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648758 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648761 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648763 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648766 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648769 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648771 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648774 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:02.653357 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648777 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648779 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648784 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648787 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648790 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648792 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648795 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648798 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648800 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648803 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648805 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648808 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648811 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648814 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648816 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648819 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648822 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648824 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648827 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648830 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:02.653864 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648833 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648835 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648838 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648842 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648846 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648849 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648852 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648854 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648857 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648860 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648862 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648865 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648867 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648870 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648873 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648877 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648881 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648885 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648888 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:02.654340 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648892 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648906 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648909 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648912 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648915 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648918 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648920 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648923 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648926 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648928 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648931 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648933 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648936 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648939 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648941 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648944 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648947 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648950 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648952 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:02.654796 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648955 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648958 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648960 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648963 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648965 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648968 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648971 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648974 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648977 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648980 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648984 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648986 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648989 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648991 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648994 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648996 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.648999 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649002 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649004 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:02.655338 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649007 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649009 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.649043 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649186 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649190 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649193 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649196 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649199 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649202 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649206 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649210 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649213 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649216 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649218 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649221 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:02.655791 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649224 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649226 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649229 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649232 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649234 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649237 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649240 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649243 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649246 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649249 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649252 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649255 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649258 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649260 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649263 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649266 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649268 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649271 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649273 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:02.656170 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649276 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649278 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649281 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649283 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649286 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649289 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649291 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649294 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649296 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649299 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649301 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649304 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649307 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649309 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649312 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649314 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649317 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649319 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649322 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649325 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:02.656617 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649328 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649331 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649334 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649337 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649340 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649342 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649345 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649347 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649350 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649352 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649355 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649358 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649360 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649363 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649366 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649368 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649371 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649373 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649376 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649379 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:02.657145 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649381 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649384 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649387 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649389 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649392 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649394 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649397 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649399 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649403 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649405 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649408 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649411 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649414 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649416 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649419 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649422 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649425 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649427 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649430 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649433 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:02.657626 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649435 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649439 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649443 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649447 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649450 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649453 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649455 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649458 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649461 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649463 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649466 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649468 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649471 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649473 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649476 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:02.658108 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.649498 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649530 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649534 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649537 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649539 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649542 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649545 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649547 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649550 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649553 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649556 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649558 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649561 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649564 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649567 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649570 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649572 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649574 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649577 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:02.658467 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649580 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649582 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649585 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649588 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649590 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649593 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649595 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649598 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649600 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649603 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649605 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649608 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649610 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649613 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649615 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649618 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649621 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649624 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649627 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649629 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:02.658918 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649632 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649634 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649637 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649640 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649642 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649645 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649647 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649650 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649660 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649663 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649666 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649669 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649671 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649674 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649676 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649679 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649681 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649685 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649687 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649690 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:02.659400 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649693 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649697 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649701 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649705 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649708 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649711 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649714 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649717 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649720 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649722 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649725 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649728 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649731 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649733 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649736 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649739 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649742 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649744 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649748 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:02.659880 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649750 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649753 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649758 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649762 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649764 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649767 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649769 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649772 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649775 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649777 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649780 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649785 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649787 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649790 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649793 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649795 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649798 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649800 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649803 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649806 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:02.660364 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649808 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649810 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649813 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649816 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649818 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649821 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649823 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649826 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:02.649829 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.649893 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.654864 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.655940 2580 server.go:1014] "Starting client certificate rotation" Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.656038 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.656060 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:22:02.660851 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.659694 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:22:02.661342 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.660941 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:22:02.671431 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.671410 2580 log.go:25] "Validated CRI v1 runtime API" Nov 01 22:22:02.675008 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.674986 2580 log.go:25] "Validated CRI v1 image API" Nov 01 22:22:02.675855 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.675831 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 22:22:02.678444 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.678414 2580 fs.go:135] Filesystem UUIDs: map[0b2edfcf-f608-4439-a77e-9461fdc3e9f4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7b6f287e-d3a8-4b7d-ac35-0cf118394913:/dev/nvme0n1p3] Nov 01 22:22:02.678535 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.678441 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 22:22:02.684793 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.684650 2580 manager.go:217] Machine: {Timestamp:2026-11-01 22:22:02.68249448 +0000 UTC m=+0.321631036 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101557 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2250801134e2a781c94211fbac68ea SystemUUID:ec225080-1134-e2a7-81c9-4211fbac68ea BootID:5255c7cb-0ed5-4d44-b1cd-bc10ea7d219b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:ce:52:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:ce:52:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:11:d2:e1:97:61 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 22:22:02.685406 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.685394 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 22:22:02.685519 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.685506 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 22:22:02.687516 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.687499 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 22:22:02.689577 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.689542 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 22:22:02.689729 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.689580 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-88.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 22:22:02.689780 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.689732 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 22:22:02.689780 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.689740 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 22:22:02.689780 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.689757 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:22:02.689780 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.689773 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:22:02.690880 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.690869 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 22:22:02.695316 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.695301 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 22:22:02.695455 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.695445 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 22:22:02.695488 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.695481 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 01 22:22:02.695514 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.695495 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 22:22:02.695514 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.695507 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 22:22:02.695581 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.695518 2580 kubelet.go:405] "Adding apiserver pod source" Nov 01 22:22:02.695581 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.695528 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 22:22:02.695972 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.695946 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:22:02.699508 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.699479 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 22:22:02.707148 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.707122 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:22:02.708135 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.708122 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 22:22:02.708185 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.708153 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 22:22:02.708325 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.708314 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 22:22:02.708373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.708330 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 22:22:02.708373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.708337 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 22:22:02.708373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.708343 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 22:22:02.708373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.708349 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 22:22:02.708373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.708355 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 22:22:02.709150 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.709129 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 22:22:02.709150 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.709146 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 22:22:02.709150 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.709154 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 22:22:02.709340 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.709161 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 22:22:02.709340 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.709175 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 22:22:02.709340 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.709184 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 22:22:02.709701 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.709679 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-88.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 22:22:02.709701 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.709680 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 22:22:02.710146 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.710129 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 22:22:02.710146 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.710145 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 22:22:02.714018 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.713985 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-88.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 22:22:02.714387 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.714375 2580 server.go:1263] "Started kubelet" Nov 01 22:22:02.714489 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.714450 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 22:22:02.715121 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.715061 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 22:22:02.715223 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.715156 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 22:22:02.715312 ip-10-0-134-88 systemd[1]: Started Kubernetes Kubelet. Nov 01 22:22:02.716533 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.716520 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 01 22:22:02.717853 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.717835 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 22:22:02.720925 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.720876 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 22:22:02.720925 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.719737 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-88.us-east-2.compute.internal.18e40bfaa392ef31 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-88.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-88.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:02.714345265 +0000 UTC m=+0.353481822,LastTimestamp:2026-11-01 22:22:02.714345265 +0000 UTC m=+0.353481822,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-88.us-east-2.compute.internal,}" Nov 01 22:22:02.723611 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.723596 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 22:22:02.724272 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.724249 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 22:22:02.724532 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.724496 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:02.725410 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.725167 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 22:22:02.725410 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.725271 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 22:22:02.725410 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.725284 2580 factory.go:55] Registering systemd factory Nov 01 22:22:02.725410 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.725292 2580 factory.go:223] Registration of the systemd container factory successfully Nov 01 22:22:02.725670 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.725498 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 22:22:02.725670 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.725515 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 22:22:02.725670 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.725651 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 01 22:22:02.725670 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.725657 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 01 22:22:02.726055 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.726041 2580 factory.go:153] Registering CRI-O factory Nov 01 22:22:02.726135 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.726058 2580 factory.go:223] Registration of the crio container factory successfully Nov 01 22:22:02.726135 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.726077 2580 factory.go:103] Registering Raw factory Nov 01 22:22:02.726135 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.726090 2580 manager.go:1199] Started watching for new ooms in manager Nov 01 22:22:02.727290 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.727267 2580 manager.go:319] Starting recovery of all containers Nov 01 22:22:02.736270 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.736234 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-88.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 22:22:02.736385 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.736324 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 01 22:22:02.739694 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.739672 2580 manager.go:324] Recovery completed Nov 01 22:22:02.744752 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.744734 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:02.746628 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.746607 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:02.746717 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.746656 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:02.746717 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.746671 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:02.747251 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.747236 2580 cpu_manager.go:226] "Starting" policy="none" Nov 01 22:22:02.747251 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.747249 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 22:22:02.747352 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.747270 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 22:22:02.747352 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.747275 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:02.748041 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.748026 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:02.748104 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.748055 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:02.748104 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.748071 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:02.749047 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.748978 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-88.us-east-2.compute.internal.18e40bfaa57f85ee default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-88.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-88.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-88.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:02.746627566 +0000 UTC m=+0.385764125,LastTimestamp:2026-11-01 22:22:02.746627566 +0000 UTC m=+0.385764125,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-88.us-east-2.compute.internal,}" Nov 01 22:22:02.749606 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.749596 2580 policy_none.go:50] "Start" Nov 01 22:22:02.749634 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.749611 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 22:22:02.749634 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.749622 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 22:22:02.751605 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.751594 2580 policy_none.go:44] "Start" Nov 01 22:22:02.757545 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.757465 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-88.us-east-2.compute.internal.18e40bfaa58011ca default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-88.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-88.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-88.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:02.74666337 +0000 UTC m=+0.385799928,LastTimestamp:2026-11-01 22:22:02.74666337 +0000 UTC m=+0.385799928,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-88.us-east-2.compute.internal,}" Nov 01 22:22:02.766550 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.766455 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-88.us-east-2.compute.internal.18e40bfaa580453c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-88.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-88.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-88.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:02.74667654 +0000 UTC m=+0.385813098,LastTimestamp:2026-11-01 22:22:02.74667654 +0000 UTC m=+0.385813098,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-88.us-east-2.compute.internal,}" Nov 01 22:22:02.770532 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.770501 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kzqj5" Nov 01 22:22:02.776490 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.776384 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-88.us-east-2.compute.internal.18e40bfaa57f85ee\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-88.us-east-2.compute.internal.18e40bfaa57f85ee default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-88.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-88.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-88.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:02.746627566 +0000 UTC m=+0.385764125,LastTimestamp:2026-11-01 22:22:02.748040803 +0000 UTC m=+0.387177364,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-88.us-east-2.compute.internal,}" Nov 01 22:22:02.778067 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.778045 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kzqj5" Nov 01 22:22:02.825184 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.825140 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:02.858734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.858696 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 22:22:02.861578 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.861558 2580 manager.go:341] "Starting Device Plugin manager" Nov 01 22:22:02.861831 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.861664 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 22:22:02.861831 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.861830 2580 server.go:87] "Starting device plugin registration server" Nov 01 22:22:02.862106 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.862092 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 22:22:02.862180 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.862103 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 22:22:02.862331 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.862224 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 22:22:02.862331 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.862306 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 22:22:02.862331 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.862315 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 22:22:02.862930 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.862893 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 22:22:02.863030 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.862961 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:02.880639 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.880603 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 22:22:02.881876 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.881852 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 22:22:02.882049 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.881882 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 22:22:02.882049 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.881924 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 22:22:02.882049 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.881969 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 22:22:02.885106 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.885087 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:02.942123 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.942083 2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-88.us-east-2.compute.internal\" not found" node="ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:02.962332 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.962305 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:02.963352 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.963329 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:02.963432 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.963365 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:02.963432 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.963380 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:02.963432 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.963407 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:02.969285 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.969258 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:02.969411 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.969292 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-88.us-east-2.compute.internal\": node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:02.982486 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.982450 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal"] Nov 01 22:22:02.982650 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.982526 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:02.983514 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.983497 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:02.983585 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.983528 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:02.983585 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.983538 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:02.984674 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:02.984656 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:02.985660 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.985648 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:02.985804 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.985791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:02.985865 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.985816 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:02.986417 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.986398 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:02.986511 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.986406 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:02.986511 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.986442 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:02.986511 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.986457 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:02.986511 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.986423 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:02.986511 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.986511 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:02.988482 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.988465 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:02.988573 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.988495 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:02.989265 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.989248 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:02.989374 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.989278 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:02.989374 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:02.989291 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:03.026700 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:03.026672 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" node="ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:03.027300 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:03.027274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:03.027403 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:03.027318 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9c6d0ee7daac240eb44f117e691eaf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"da9c6d0ee7daac240eb44f117e691eaf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:03.027403 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:03.027338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:03.029461 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:03.029442 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" node="ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:03.084844 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:03.084810 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:03.128161 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:03.128124 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:03.128161 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:03.128162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9c6d0ee7daac240eb44f117e691eaf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"da9c6d0ee7daac240eb44f117e691eaf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:03.128410 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:03.128179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:03.128410 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:03.128210 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9c6d0ee7daac240eb44f117e691eaf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"da9c6d0ee7daac240eb44f117e691eaf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:03.128410 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:03.128215 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:03.128410 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:03.128243 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:03.185990 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:03.185883 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:04.398792 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:04.398758 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:04.442280 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:04.442240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:04.445357 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:04.445339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:04.499463 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:04.499427 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:04.599982 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:04.599904 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:04.700497 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:04.700464 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:04.767953 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:04.767920 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 22:22:04.768458 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:04.768098 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 22:22:04.801249 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:04.801215 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:04.895801 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:04.895714 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:03 +0000 UTC" deadline="2028-04-03 07:17:11.624385167 +0000 UTC" Nov 01 22:22:04.895801 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:04.895747 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12440h55m6.728641659s" Nov 01 22:22:04.902001 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:04.901985 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:04.906290 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:04.906267 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:05.002638 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.002612 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:05.011771 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.011750 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:05.014194 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.014176 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:05.014286 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.014204 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:05.014286 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.014216 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:05.103762 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.103730 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:05.135172 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:05.135128 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615e11a436a671e663c9b9217b7e9606.slice/crio-df30684be22fdfc276dbba87c25d24c6e68907c5da7ff9184d651db76991a3f1 WatchSource:0}: Error finding container df30684be22fdfc276dbba87c25d24c6e68907c5da7ff9184d651db76991a3f1: Status 404 returned error can't find the container with id df30684be22fdfc276dbba87c25d24c6e68907c5da7ff9184d651db76991a3f1 Nov 01 22:22:05.135742 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:05.135718 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9c6d0ee7daac240eb44f117e691eaf.slice/crio-5b75527010af335f6bc91af969b0d3f5a9ec9ae1caf1ec6b95d5c1224420b98f WatchSource:0}: Error finding container 5b75527010af335f6bc91af969b0d3f5a9ec9ae1caf1ec6b95d5c1224420b98f: Status 404 returned error can't find the container with id 5b75527010af335f6bc91af969b0d3f5a9ec9ae1caf1ec6b95d5c1224420b98f Nov 01 22:22:05.141817 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.141705 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:22:05.204694 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.204620 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:05.247262 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.247234 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:05.305483 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.305454 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:05.406200 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.406170 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:22:05.409599 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.409584 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:05.437631 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.437607 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:05.448209 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.448189 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:22:05.449018 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.449005 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:22:05.456898 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.456845 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:22:05.809526 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.809457 2580 apiserver.go:52] "Watching apiserver" Nov 01 22:22:05.816450 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.816426 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 22:22:05.816796 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.816771 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-8cbk8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-cg4rv","openshift-multus/multus-additional-cni-plugins-t8rz8","openshift-network-diagnostics/network-check-target-vzk9n","kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-f4rmp","openshift-image-registry/node-ca-f9snb","openshift-network-operator/iptables-alerter-v2bl8","openshift-ovn-kubernetes/ovnkube-node-zntdq","kube-system/konnectivity-agent-fxlz9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt","openshift-dns/node-resolver-mfnxk","openshift-multus/multus-qj4tr"] Nov 01 22:22:05.820836 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.820820 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.821401 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.821375 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:22:05.823718 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.823696 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:05.823812 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.823776 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:05.824841 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.824822 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dclh4\"" Nov 01 22:22:05.824936 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.824859 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:22:05.825203 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.825183 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 22:22:05.825203 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.825186 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 22:22:05.825340 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.825218 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 22:22:05.825340 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.825268 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-79k59\"" Nov 01 22:22:05.825864 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.825845 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.828065 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.828049 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:05.828171 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.828126 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:05.830620 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.830600 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.832846 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.832821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:22:05.835779 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.835747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:05.835898 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.835807 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:05.837000 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.836966 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 22:22:05.838769 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.838749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:22:05.841880 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.841352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.844199 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.844179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:05.846428 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.846410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:05.849482 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:05.849586 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-lib-modules\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.849586 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.849586 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:05.849740 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-netns\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.849740 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-bin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.849740 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849684 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-kubelet\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.849894 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-multus-certs\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.849894 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-iptables-alerter-script\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:22:05.849894 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-host-slash\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:22:05.850047 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysconfig\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.850047 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-sys\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.850047 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-var-lib-kubelet\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.850047 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.849992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-system-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.850047 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-cnibin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.850047 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-modprobe-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-systemd\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-dbus\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-multus\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-conf\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-host\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a5a648-0585-408d-8556-c0e31dcea942-tmp-dir\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsfjt\" (UniqueName: \"kubernetes.io/projected/6962a1b2-43de-4da8-a1f4-af8101d42fd0-kube-api-access-lsfjt\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850275 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-k8s-cni-cncf-io\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-hostroot\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-etc-kubernetes\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrbgq\" (UniqueName: \"kubernetes.io/projected/5baeb178-3616-4b1d-8515-d8ff71486666-kube-api-access-qrbgq\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.850373 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-tuned\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.850907 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6962a1b2-43de-4da8-a1f4-af8101d42fd0-host\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:22:05.850907 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.850513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6962a1b2-43de-4da8-a1f4-af8101d42fd0-serviceca\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:22:05.851324 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-system-cni-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.851441 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wfck\" (UniqueName: \"kubernetes.io/projected/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-kube-api-access-9wfck\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:05.851441 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-tmp\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.851546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-cnibin\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.851546 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851528 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-binary-copy\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.851651 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.851651 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7sqc\" (UniqueName: \"kubernetes.io/projected/4d408013-e118-42d1-809f-5d3b50f12565-kube-api-access-j7sqc\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.851651 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.851792 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851663 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 22:22:05.851792 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-conf-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.851792 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5dtj\" (UniqueName: \"kubernetes.io/projected/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-kube-api-access-k5dtj\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:22:05.851792 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-run\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.851952 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76a5a648-0585-408d-8556-c0e31dcea942-hosts-file\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:22:05.851952 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hsxj\" (UniqueName: \"kubernetes.io/projected/76a5a648-0585-408d-8556-c0e31dcea942-kube-api-access-2hsxj\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:22:05.851952 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-kubelet-config\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:05.851952 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.851952 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-socket-dir-parent\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.852207 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.851965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-multus-daemon-config\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.852207 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.852045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-kubernetes\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.852207 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.852076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6gp\" (UniqueName: \"kubernetes.io/projected/f2c4e933-b200-47a6-94c8-1966aeb16884-kube-api-access-nt6gp\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.852207 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.852131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:05.852207 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.852162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-cni-binary-copy\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.864682 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.864657 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mktxn" Nov 01 22:22:05.872047 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.872025 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mktxn" Nov 01 22:22:05.924643 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.924589 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 22:22:05.924853 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.924812 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 22:22:05.925070 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.925048 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4hpp8\"" Nov 01 22:22:05.925070 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.925057 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p26bd\"" Nov 01 22:22:05.925975 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.925957 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 22:22:05.926045 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.925989 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vvcmq\"" Nov 01 22:22:05.926230 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926193 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 22:22:05.926230 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926224 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 22:22:05.926451 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926278 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zwl5x\"" Nov 01 22:22:05.926451 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926349 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j5zd8\"" Nov 01 22:22:05.926451 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926378 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 22:22:05.926642 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926492 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 22:22:05.926642 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926517 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qjk9v\"" Nov 01 22:22:05.926642 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926540 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 22:22:05.926642 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926567 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 22:22:05.926642 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926605 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 22:22:05.926642 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926629 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 22:22:05.926642 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926649 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 22:22:05.926968 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926662 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 22:22:05.926968 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926667 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vv5lb\"" Nov 01 22:22:05.926968 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926697 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 22:22:05.926968 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926709 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 22:22:05.926968 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926719 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 22:22:05.926968 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926721 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 22:22:05.926968 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.926747 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:22:05.927493 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.927468 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 22:22:05.927493 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.927474 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 22:22:05.927630 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.927506 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 22:22:05.927630 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.927539 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 22:22:05.927948 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.927927 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 22:22:05.938338 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.938319 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 22:22:05.953534 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.953505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-node-log\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.953670 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.953556 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-log-socket\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.953670 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.953593 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-netd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.953670 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.953625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-run\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.953813 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.953734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hsxj\" (UniqueName: \"kubernetes.io/projected/76a5a648-0585-408d-8556-c0e31dcea942-kube-api-access-2hsxj\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:22:05.953867 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.953826 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-run\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.953915 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.953896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-socket-dir-parent\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.953964 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.953934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-multus-daemon-config\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.954010 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.953967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-kubernetes\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.954063 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.953999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:05.954063 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.954055 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-socket-dir-parent\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.954230 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.954211 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:05.954291 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.954247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.954331 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.954308 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret podName:8739003e-fba1-4d47-80e9-0f484d117b73 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:06.454267458 +0000 UTC m=+2.981195215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret") pod "global-pull-secret-syncer-cg4rv" (UID: "8739003e-fba1-4d47-80e9-0f484d117b73") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:05.954399 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.954339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-script-lib\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.954399 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.954380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:05.954469 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.954430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-netns\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.954515 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.954475 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:05.954515 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.954183 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-kubernetes\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.954911 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.954888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-bin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.955002 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.954919 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-netns\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.955002 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.954972 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs podName:b21a7a1c-6db6-422d-a24f-5bdc6b4a700c nodeName:}" failed. No retries permitted until 2026-11-01 22:22:06.454951936 +0000 UTC m=+2.981879695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs") pod "network-metrics-daemon-8cbk8" (UID: "b21a7a1c-6db6-422d-a24f-5bdc6b4a700c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:05.955138 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-kubelet\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.955138 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955032 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-bin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.955138 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.955138 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955076 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-kubelet\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.955138 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-registration-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:05.955405 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-sys-fs\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:05.955405 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955158 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-multus-daemon-config\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.955405 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-cnibin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.955405 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955251 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-cnibin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.955405 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-bin\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.955405 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-modprobe-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.955405 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-systemd\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.955405 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955362 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-modprobe-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.955405 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955406 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-systemd\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-dbus\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-systemd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-conf\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsfjt\" (UniqueName: \"kubernetes.io/projected/6962a1b2-43de-4da8-a1f4-af8101d42fd0-kube-api-access-lsfjt\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-k8s-cni-cncf-io\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-ovn\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-tuned\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955592 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-dbus\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6962a1b2-43de-4da8-a1f4-af8101d42fd0-host\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955616 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-conf\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-system-cni-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-socket-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:05.955734 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955733 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-k8s-cni-cncf-io\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955737 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6962a1b2-43de-4da8-a1f4-af8101d42fd0-host\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-etc-selinux\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955783 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-system-cni-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkns2\" (UniqueName: \"kubernetes.io/projected/ea837196-594f-42a6-981b-74717f55e790-kube-api-access-qkns2\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-multus-certs\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-etc-kubernetes\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wfck\" (UniqueName: \"kubernetes.io/projected/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-kube-api-access-9wfck\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.955993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-tmp\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956055 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-etc-kubernetes\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956083 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-multus-certs\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-systemd-units\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.956205 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956165 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5dtj\" (UniqueName: \"kubernetes.io/projected/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-kube-api-access-k5dtj\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76a5a648-0585-408d-8556-c0e31dcea942-hosts-file\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-kubelet-config\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956310 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76a5a648-0585-408d-8556-c0e31dcea942-hosts-file\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-device-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956440 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-kubelet-config\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6gp\" (UniqueName: \"kubernetes.io/projected/f2c4e933-b200-47a6-94c8-1966aeb16884-kube-api-access-nt6gp\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-cni-binary-copy\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956531 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956546 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-slash\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.956777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-lib-modules\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956918 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-lib-modules\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-kubelet\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.956980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-netns\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957052 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-cni-binary-copy\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-var-lib-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-config\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovn-node-metrics-cert\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-iptables-alerter-script\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:22:05.957327 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-host-slash\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysconfig\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-sys\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957446 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-var-lib-kubelet\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-system-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-env-overrides\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957546 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysconfig\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957587 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-var-lib-kubelet\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-multus\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957628 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-system-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1baadc8-f14c-490f-9655-a55352e7eb8b-agent-certs\") pod \"konnectivity-agent-fxlz9\" (UID: \"e1baadc8-f14c-490f-9655-a55352e7eb8b\") " pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-host\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a5a648-0585-408d-8556-c0e31dcea942-tmp-dir\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957700 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-iptables-alerter-script\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957700 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-sys\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.957807 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-hostroot\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957502 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-host-slash\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957778 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-hostroot\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957795 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-multus\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrbgq\" (UniqueName: \"kubernetes.io/projected/5baeb178-3616-4b1d-8515-d8ff71486666-kube-api-access-qrbgq\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957870 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1baadc8-f14c-490f-9655-a55352e7eb8b-konnectivity-ca\") pod \"konnectivity-agent-fxlz9\" (UID: \"e1baadc8-f14c-490f-9655-a55352e7eb8b\") " pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957935 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-host\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6962a1b2-43de-4da8-a1f4-af8101d42fd0-serviceca\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.957982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-etc-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.958040 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a5a648-0585-408d-8556-c0e31dcea942-tmp-dir\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.958091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjpf6\" (UniqueName: \"kubernetes.io/projected/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-kube-api-access-qjpf6\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.958268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-cnibin\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.958336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-binary-copy\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.958342 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-cnibin\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.958380 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6962a1b2-43de-4da8-a1f4-af8101d42fd0-serviceca\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.958376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.958644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.958419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7sqc\" (UniqueName: \"kubernetes.io/projected/4d408013-e118-42d1-809f-5d3b50f12565-kube-api-access-j7sqc\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.959456 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.958446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-conf-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.959456 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.958551 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-conf-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.959456 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.958886 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.959456 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.959289 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-binary-copy\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.960266 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.960237 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-tmp\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.960394 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.960265 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-tuned\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.964277 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.964255 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hsxj\" (UniqueName: \"kubernetes.io/projected/76a5a648-0585-408d-8556-c0e31dcea942-kube-api-access-2hsxj\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:22:05.967159 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.967140 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:05.967251 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.967163 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:05.967251 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.967175 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7w5t7 for pod openshift-network-diagnostics/network-check-target-vzk9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:05.967251 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:05.967223 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7 podName:4dbe2700-733d-4ebd-9f47-7b9f30bb62fd nodeName:}" failed. No retries permitted until 2026-11-01 22:22:06.467208105 +0000 UTC m=+2.994135864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7w5t7" (UniqueName: "kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7") pod "network-check-target-vzk9n" (UID: "4dbe2700-733d-4ebd-9f47-7b9f30bb62fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:05.969699 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.969673 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsfjt\" (UniqueName: \"kubernetes.io/projected/6962a1b2-43de-4da8-a1f4-af8101d42fd0-kube-api-access-lsfjt\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:22:05.969699 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.969690 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5dtj\" (UniqueName: \"kubernetes.io/projected/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-kube-api-access-k5dtj\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:22:05.970122 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.970088 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrbgq\" (UniqueName: \"kubernetes.io/projected/5baeb178-3616-4b1d-8515-d8ff71486666-kube-api-access-qrbgq\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:22:05.970811 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.970772 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7sqc\" (UniqueName: \"kubernetes.io/projected/4d408013-e118-42d1-809f-5d3b50f12565-kube-api-access-j7sqc\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:05.971020 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.971002 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wfck\" (UniqueName: \"kubernetes.io/projected/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-kube-api-access-9wfck\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:05.972236 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.972218 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6gp\" (UniqueName: \"kubernetes.io/projected/f2c4e933-b200-47a6-94c8-1966aeb16884-kube-api-access-nt6gp\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:05.999594 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:05.999551 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" event={"ID":"615e11a436a671e663c9b9217b7e9606","Type":"ContainerStarted","Data":"df30684be22fdfc276dbba87c25d24c6e68907c5da7ff9184d651db76991a3f1"} Nov 01 22:22:06.000761 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.000733 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" event={"ID":"da9c6d0ee7daac240eb44f117e691eaf","Type":"ContainerStarted","Data":"5b75527010af335f6bc91af969b0d3f5a9ec9ae1caf1ec6b95d5c1224420b98f"} Nov 01 22:22:06.059809 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-systemd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.059809 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-ovn\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.059809 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-socket-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-etc-selinux\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059846 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-ovn\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059850 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-systemd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkns2\" (UniqueName: \"kubernetes.io/projected/ea837196-594f-42a6-981b-74717f55e790-kube-api-access-qkns2\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-systemd-units\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-device-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059968 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-etc-selinux\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-slash\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.059983 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-socket-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060033 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-device-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060036 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-slash\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-kubelet\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060062 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060043 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-systemd-units\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060073 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-netns\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060086 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-kubelet\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060103 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-netns\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060140 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-var-lib-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-config\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovn-node-metrics-cert\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-env-overrides\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060222 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-var-lib-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1baadc8-f14c-490f-9655-a55352e7eb8b-agent-certs\") pod \"konnectivity-agent-fxlz9\" (UID: \"e1baadc8-f14c-490f-9655-a55352e7eb8b\") " pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060547 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1baadc8-f14c-490f-9655-a55352e7eb8b-konnectivity-ca\") pod \"konnectivity-agent-fxlz9\" (UID: \"e1baadc8-f14c-490f-9655-a55352e7eb8b\") " pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:06.060638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-etc-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjpf6\" (UniqueName: \"kubernetes.io/projected/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-kube-api-access-qjpf6\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-node-log\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-log-socket\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-netd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-script-lib\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060810 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-env-overrides\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060849 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-config\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060888 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-log-socket\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060896 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060944 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-etc-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.060984 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-netd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.061011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-registration-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.061037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-sys-fs\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.061063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-bin\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.061372 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.061146 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-bin\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.062202 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.061160 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1baadc8-f14c-490f-9655-a55352e7eb8b-konnectivity-ca\") pod \"konnectivity-agent-fxlz9\" (UID: \"e1baadc8-f14c-490f-9655-a55352e7eb8b\") " pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:06.062202 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.061173 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-node-log\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.062202 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.061196 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-registration-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.062202 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.061216 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.062202 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.061230 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-sys-fs\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.062202 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.061388 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-script-lib\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.062808 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.062784 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovn-node-metrics-cert\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.063882 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.063858 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1baadc8-f14c-490f-9655-a55352e7eb8b-agent-certs\") pod \"konnectivity-agent-fxlz9\" (UID: \"e1baadc8-f14c-490f-9655-a55352e7eb8b\") " pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:06.067980 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.067957 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkns2\" (UniqueName: \"kubernetes.io/projected/ea837196-594f-42a6-981b-74717f55e790-kube-api-access-qkns2\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.068292 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.068267 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjpf6\" (UniqueName: \"kubernetes.io/projected/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-kube-api-access-qjpf6\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.134259 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.134218 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:22:06.144162 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.144136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:22:06.145212 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:06.145130 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c4e933_b200_47a6_94c8_1966aeb16884.slice/crio-f0d9f506c5d155d467d20cb76c0598bece48bf923f426878e585afe5d546f66f WatchSource:0}: Error finding container f0d9f506c5d155d467d20cb76c0598bece48bf923f426878e585afe5d546f66f: Status 404 returned error can't find the container with id f0d9f506c5d155d467d20cb76c0598bece48bf923f426878e585afe5d546f66f Nov 01 22:22:06.153699 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:06.153674 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a5a648_0585_408d_8556_c0e31dcea942.slice/crio-f2c80714eec266084edc75df0ba27e03ab2ab3216ff8d03ee43f1343555934e5 WatchSource:0}: Error finding container f2c80714eec266084edc75df0ba27e03ab2ab3216ff8d03ee43f1343555934e5: Status 404 returned error can't find the container with id f2c80714eec266084edc75df0ba27e03ab2ab3216ff8d03ee43f1343555934e5 Nov 01 22:22:06.168300 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.168276 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qj4tr" Nov 01 22:22:06.169341 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.169315 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:22:06.171466 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.171445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:22:06.176912 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.176842 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:22:06.184004 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:06.183980 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5baeb178_3616_4b1d_8515_d8ff71486666.slice/crio-c31bacab52a6216b2f7a96ce3d9b44efe0246fcfd576ca4c35a77be8565ac4b6 WatchSource:0}: Error finding container c31bacab52a6216b2f7a96ce3d9b44efe0246fcfd576ca4c35a77be8565ac4b6: Status 404 returned error can't find the container with id c31bacab52a6216b2f7a96ce3d9b44efe0246fcfd576ca4c35a77be8565ac4b6 Nov 01 22:22:06.189042 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:06.188864 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1dd1e50_ecd0_4724_862d_752e9e1aecd0.slice/crio-98cd666dca399b17251fe1448ac855173ee885cc270031f12268dfb12b10bad9 WatchSource:0}: Error finding container 98cd666dca399b17251fe1448ac855173ee885cc270031f12268dfb12b10bad9: Status 404 returned error can't find the container with id 98cd666dca399b17251fe1448ac855173ee885cc270031f12268dfb12b10bad9 Nov 01 22:22:06.193148 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:06.193122 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6962a1b2_43de_4da8_a1f4_af8101d42fd0.slice/crio-2aff8080095aa44f043614b07c3f72b0d828a4c9db426dd5b46847e31f12daf2 WatchSource:0}: Error finding container 2aff8080095aa44f043614b07c3f72b0d828a4c9db426dd5b46847e31f12daf2: Status 404 returned error can't find the container with id 2aff8080095aa44f043614b07c3f72b0d828a4c9db426dd5b46847e31f12daf2 Nov 01 22:22:06.200322 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.200301 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:06.209870 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.209850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:06.210362 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:06.210326 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1baadc8_f14c_490f_9655_a55352e7eb8b.slice/crio-c478781b15d1eddbed88055f52b54ff23b89ea0b3c980619a6d32e795b310fc6 WatchSource:0}: Error finding container c478781b15d1eddbed88055f52b54ff23b89ea0b3c980619a6d32e795b310fc6: Status 404 returned error can't find the container with id c478781b15d1eddbed88055f52b54ff23b89ea0b3c980619a6d32e795b310fc6 Nov 01 22:22:06.211086 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.210924 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:22:06.221150 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:06.221127 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5b3c1d_6e53_422e_b15a_ca0b66d4dd2c.slice/crio-288cef5dfe34637db0488d39d62c0703785a30b8abd5ab890a2a54d4254c5097 WatchSource:0}: Error finding container 288cef5dfe34637db0488d39d62c0703785a30b8abd5ab890a2a54d4254c5097: Status 404 returned error can't find the container with id 288cef5dfe34637db0488d39d62c0703785a30b8abd5ab890a2a54d4254c5097 Nov 01 22:22:06.304893 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.304858 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:06.465644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.465572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:06.465644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.465614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:06.465857 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:06.465743 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:06.465857 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:06.465762 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:06.465857 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:06.465815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs podName:b21a7a1c-6db6-422d-a24f-5bdc6b4a700c nodeName:}" failed. No retries permitted until 2026-11-01 22:22:07.465799226 +0000 UTC m=+3.992726987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs") pod "network-metrics-daemon-8cbk8" (UID: "b21a7a1c-6db6-422d-a24f-5bdc6b4a700c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:06.465857 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:06.465835 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret podName:8739003e-fba1-4d47-80e9-0f484d117b73 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:07.46582498 +0000 UTC m=+3.992752728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret") pod "global-pull-secret-syncer-cg4rv" (UID: "8739003e-fba1-4d47-80e9-0f484d117b73") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:06.566858 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.566823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:06.567033 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:06.566991 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:06.567033 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:06.567011 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:06.567033 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:06.567021 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7w5t7 for pod openshift-network-diagnostics/network-check-target-vzk9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:06.567217 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:06.567084 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7 podName:4dbe2700-733d-4ebd-9f47-7b9f30bb62fd nodeName:}" failed. No retries permitted until 2026-11-01 22:22:07.567063889 +0000 UTC m=+4.093991649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w5t7" (UniqueName: "kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7") pod "network-check-target-vzk9n" (UID: "4dbe2700-733d-4ebd-9f47-7b9f30bb62fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:06.874966 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.874852 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:05 +0000 UTC" deadline="2028-07-17 05:13:36.221219286 +0000 UTC" Nov 01 22:22:06.874966 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.874919 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14958h51m29.346304338s" Nov 01 22:22:06.994809 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:06.994364 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:06.994809 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:06.994477 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:07.007230 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.007193 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" event={"ID":"ea837196-594f-42a6-981b-74717f55e790","Type":"ContainerStarted","Data":"0b8931f969e4f432f5b9512dc174708eefdf8a4e50e264ee70b00cb41e3bd8a6"} Nov 01 22:22:07.019501 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.019364 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fxlz9" event={"ID":"e1baadc8-f14c-490f-9655-a55352e7eb8b","Type":"ContainerStarted","Data":"c478781b15d1eddbed88055f52b54ff23b89ea0b3c980619a6d32e795b310fc6"} Nov 01 22:22:07.028325 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.028220 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" event={"ID":"4d408013-e118-42d1-809f-5d3b50f12565","Type":"ContainerStarted","Data":"eda1485c80c9159bf77bb63e96a59ce816c02b7f3b84c283e36cdcb5595df434"} Nov 01 22:22:07.031829 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.031577 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" event={"ID":"f2c4e933-b200-47a6-94c8-1966aeb16884","Type":"ContainerStarted","Data":"f0d9f506c5d155d467d20cb76c0598bece48bf923f426878e585afe5d546f66f"} Nov 01 22:22:07.034510 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.034487 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" event={"ID":"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c","Type":"ContainerStarted","Data":"288cef5dfe34637db0488d39d62c0703785a30b8abd5ab890a2a54d4254c5097"} Nov 01 22:22:07.036220 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.036194 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f9snb" event={"ID":"6962a1b2-43de-4da8-a1f4-af8101d42fd0","Type":"ContainerStarted","Data":"2aff8080095aa44f043614b07c3f72b0d828a4c9db426dd5b46847e31f12daf2"} Nov 01 22:22:07.041521 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.041493 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v2bl8" event={"ID":"e1dd1e50-ecd0-4724-862d-752e9e1aecd0","Type":"ContainerStarted","Data":"98cd666dca399b17251fe1448ac855173ee885cc270031f12268dfb12b10bad9"} Nov 01 22:22:07.043826 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.043740 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qj4tr" event={"ID":"5baeb178-3616-4b1d-8515-d8ff71486666","Type":"ContainerStarted","Data":"c31bacab52a6216b2f7a96ce3d9b44efe0246fcfd576ca4c35a77be8565ac4b6"} Nov 01 22:22:07.057444 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.057410 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mfnxk" event={"ID":"76a5a648-0585-408d-8556-c0e31dcea942","Type":"ContainerStarted","Data":"f2c80714eec266084edc75df0ba27e03ab2ab3216ff8d03ee43f1343555934e5"} Nov 01 22:22:07.473471 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.473436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:07.473647 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.473483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:07.473647 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:07.473619 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:07.473750 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:07.473680 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs podName:b21a7a1c-6db6-422d-a24f-5bdc6b4a700c nodeName:}" failed. No retries permitted until 2026-11-01 22:22:09.473659656 +0000 UTC m=+6.000587414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs") pod "network-metrics-daemon-8cbk8" (UID: "b21a7a1c-6db6-422d-a24f-5bdc6b4a700c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:07.475576 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:07.474060 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:07.475576 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:07.474122 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret podName:8739003e-fba1-4d47-80e9-0f484d117b73 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:09.474095915 +0000 UTC m=+6.001023674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret") pod "global-pull-secret-syncer-cg4rv" (UID: "8739003e-fba1-4d47-80e9-0f484d117b73") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:07.574043 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.574007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:07.574226 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:07.574202 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:07.574226 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:07.574223 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:07.574326 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:07.574236 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7w5t7 for pod openshift-network-diagnostics/network-check-target-vzk9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:07.574326 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:07.574289 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7 podName:4dbe2700-733d-4ebd-9f47-7b9f30bb62fd nodeName:}" failed. No retries permitted until 2026-11-01 22:22:09.574272101 +0000 UTC m=+6.101199848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w5t7" (UniqueName: "kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7") pod "network-check-target-vzk9n" (UID: "4dbe2700-733d-4ebd-9f47-7b9f30bb62fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:07.875757 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.875676 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:05 +0000 UTC" deadline="2028-06-10 18:04:18.679359568 +0000 UTC" Nov 01 22:22:07.875757 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.875717 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14083h42m10.803647021s" Nov 01 22:22:07.997153 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.997059 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:07.997300 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:07.997235 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:07.997756 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:07.997661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:07.997874 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:07.997797 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:08.082839 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:08.082010 2580 generic.go:356] "Generic (PLEG): container finished" podID="615e11a436a671e663c9b9217b7e9606" containerID="edcec4ee6b53796f73a7d1705c13b0dac0aad245a6d297e20a1de6571a358aef" exitCode=0 Nov 01 22:22:08.082839 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:08.082069 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" event={"ID":"615e11a436a671e663c9b9217b7e9606","Type":"ContainerDied","Data":"edcec4ee6b53796f73a7d1705c13b0dac0aad245a6d297e20a1de6571a358aef"} Nov 01 22:22:08.995784 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:08.995260 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:08.995784 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:08.995385 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:09.094133 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:09.093342 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="81c9e2fc629edb4a77c67bdaf27e4b7b66d3379d9fb62381d43ecfc877287202" exitCode=0 Nov 01 22:22:09.094133 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:09.093435 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" event={"ID":"4d408013-e118-42d1-809f-5d3b50f12565","Type":"ContainerDied","Data":"81c9e2fc629edb4a77c67bdaf27e4b7b66d3379d9fb62381d43ecfc877287202"} Nov 01 22:22:09.102840 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:09.102403 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" event={"ID":"da9c6d0ee7daac240eb44f117e691eaf","Type":"ContainerStarted","Data":"83b103e1ff219a7da8deb08d0f030f273cbdc6ce5dc4714220fbbb799f294a3e"} Nov 01 22:22:09.105631 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:09.105588 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" event={"ID":"615e11a436a671e663c9b9217b7e9606","Type":"ContainerStarted","Data":"74c446ffefd716924f6845cc5582c75daefa293c851c182a8fcf98ee4c0a4ce4"} Nov 01 22:22:09.143056 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:09.142456 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" podStartSLOduration=4.14244111 podStartE2EDuration="4.14244111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:09.128756719 +0000 UTC m=+5.655684484" watchObservedRunningTime="2026-11-01 22:22:09.14244111 +0000 UTC m=+5.669368876" Nov 01 22:22:09.495423 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:09.495383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:09.495621 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:09.495436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:09.495621 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:09.495566 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:09.495745 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:09.495631 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs podName:b21a7a1c-6db6-422d-a24f-5bdc6b4a700c nodeName:}" failed. No retries permitted until 2026-11-01 22:22:13.495614404 +0000 UTC m=+10.022542164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs") pod "network-metrics-daemon-8cbk8" (UID: "b21a7a1c-6db6-422d-a24f-5bdc6b4a700c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:09.496043 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:09.496021 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:09.496163 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:09.496075 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret podName:8739003e-fba1-4d47-80e9-0f484d117b73 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:13.496060911 +0000 UTC m=+10.022988670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret") pod "global-pull-secret-syncer-cg4rv" (UID: "8739003e-fba1-4d47-80e9-0f484d117b73") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:09.596588 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:09.595923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:09.596588 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:09.596133 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:09.596588 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:09.596158 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:09.596588 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:09.596173 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7w5t7 for pod openshift-network-diagnostics/network-check-target-vzk9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:09.596588 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:09.596226 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7 podName:4dbe2700-733d-4ebd-9f47-7b9f30bb62fd nodeName:}" failed. No retries permitted until 2026-11-01 22:22:13.59621191 +0000 UTC m=+10.123139655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w5t7" (UniqueName: "kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7") pod "network-check-target-vzk9n" (UID: "4dbe2700-733d-4ebd-9f47-7b9f30bb62fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:09.996295 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:09.996231 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:09.996866 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:09.996377 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:09.996866 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:09.996773 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:09.997007 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:09.996866 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:10.994865 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:10.994376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:10.994865 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:10.994511 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:11.122317 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:11.122278 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fxlz9" event={"ID":"e1baadc8-f14c-490f-9655-a55352e7eb8b","Type":"ContainerStarted","Data":"8438f80fd304c3bbc92cae5f189e2e633605e2fe38ccc3187c2a096a27f65de3"} Nov 01 22:22:11.142946 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:11.142885 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" podStartSLOduration=6.142862454 podStartE2EDuration="6.142862454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:09.142997511 +0000 UTC m=+5.669925278" watchObservedRunningTime="2026-11-01 22:22:11.142862454 +0000 UTC m=+7.669790257" Nov 01 22:22:12.008704 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:12.006257 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:12.008704 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:12.006505 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:12.008704 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:12.006657 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:12.008704 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:12.006776 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:12.133682 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:12.133645 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" event={"ID":"ea837196-594f-42a6-981b-74717f55e790","Type":"ContainerStarted","Data":"62343cd7ab3834fd439be355b79677341264c7a3f5c38b47bed067eed9919a56"} Nov 01 22:22:12.995239 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:12.995205 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:12.995418 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:12.995346 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:13.141171 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:13.138362 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f9snb" event={"ID":"6962a1b2-43de-4da8-a1f4-af8101d42fd0","Type":"ContainerStarted","Data":"d5d081b196a2fbc497362e9ad6403ce467b17a93edff66b10433ae6555640ff1"} Nov 01 22:22:13.153343 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:13.153255 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fxlz9" podStartSLOduration=5.456419131 podStartE2EDuration="9.15323814s" totalImagesPullingTime="3.696819009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:11.143842091 +0000 UTC m=+7.670769857" watchObservedRunningTime="2026-11-01 22:22:13.15323814 +0000 UTC m=+9.680165907" Nov 01 22:22:13.521229 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:13.521196 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:22:13.531666 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:13.531603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:13.531666 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:13.531650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:13.531812 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:13.531727 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:13.531812 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:13.531768 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:13.531812 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:13.531786 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret podName:8739003e-fba1-4d47-80e9-0f484d117b73 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:21.531770883 +0000 UTC m=+18.058698650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret") pod "global-pull-secret-syncer-cg4rv" (UID: "8739003e-fba1-4d47-80e9-0f484d117b73") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:13.531928 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:13.531818 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs podName:b21a7a1c-6db6-422d-a24f-5bdc6b4a700c nodeName:}" failed. No retries permitted until 2026-11-01 22:22:21.53180219 +0000 UTC m=+18.058729939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs") pod "network-metrics-daemon-8cbk8" (UID: "b21a7a1c-6db6-422d-a24f-5bdc6b4a700c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:13.632705 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:13.631948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:13.632705 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:13.632184 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:13.632705 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:13.632205 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:13.632705 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:13.632218 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7w5t7 for pod openshift-network-diagnostics/network-check-target-vzk9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:13.632705 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:13.632273 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7 podName:4dbe2700-733d-4ebd-9f47-7b9f30bb62fd nodeName:}" failed. No retries permitted until 2026-11-01 22:22:21.63225601 +0000 UTC m=+18.159183769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w5t7" (UniqueName: "kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7") pod "network-check-target-vzk9n" (UID: "4dbe2700-733d-4ebd-9f47-7b9f30bb62fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:13.979887 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:13.979769 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:22:13.521222849Z","UUID":"bec94289-6568-4ee9-90cc-c015954c54f1","Handler":null,"Name":"","Endpoint":""} Nov 01 22:22:13.986298 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:13.986276 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 22:22:13.986405 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:13.986310 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:22:13.997483 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:13.997446 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:13.997595 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:13.997570 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:13.997930 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:13.997912 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:13.998041 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:13.998021 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:14.146772 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:14.146538 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qj4tr" event={"ID":"5baeb178-3616-4b1d-8515-d8ff71486666","Type":"ContainerStarted","Data":"88c113de26761da763d9048894b250e1688d9d1517a431f8b3d8d11d2a420d9d"} Nov 01 22:22:14.153750 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:14.153707 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" event={"ID":"ea837196-594f-42a6-981b-74717f55e790","Type":"ContainerStarted","Data":"ae260c01eca824d6eae0b6a31c14a9741e87fc8061cb4fe8687229af778379c6"} Nov 01 22:22:14.158927 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:14.158891 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" event={"ID":"4d408013-e118-42d1-809f-5d3b50f12565","Type":"ContainerStarted","Data":"3d70c2e948fd114135ed186142d485e86353941db92d968ad2146aa3779db9f4"} Nov 01 22:22:14.161726 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:14.161636 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f9snb" podStartSLOduration=4.275260689 podStartE2EDuration="10.161617642s" totalImagesPullingTime="5.886356953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:13.154170191 +0000 UTC m=+9.681097959" watchObservedRunningTime="2026-11-01 22:22:14.161617642 +0000 UTC m=+10.688545408" Nov 01 22:22:14.162593 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:14.162553 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qj4tr" podStartSLOduration=2.33317488 podStartE2EDuration="10.162536534s" totalImagesPullingTime="7.829361654s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:14.161928203 +0000 UTC m=+10.688855951" watchObservedRunningTime="2026-11-01 22:22:14.162536534 +0000 UTC m=+10.689464301" Nov 01 22:22:14.994630 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:14.994552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:14.994792 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:14.994688 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:15.162537 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:15.162496 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" event={"ID":"ea837196-594f-42a6-981b-74717f55e790","Type":"ContainerStarted","Data":"e3bac1d0659fd9b872ff37d3cdb073efedbc3d560960519a5996a312a78c1855"} Nov 01 22:22:15.164490 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:15.164452 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="3d70c2e948fd114135ed186142d485e86353941db92d968ad2146aa3779db9f4" exitCode=0 Nov 01 22:22:15.164613 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:15.164546 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" event={"ID":"4d408013-e118-42d1-809f-5d3b50f12565","Type":"ContainerDied","Data":"3d70c2e948fd114135ed186142d485e86353941db92d968ad2146aa3779db9f4"} Nov 01 22:22:15.167407 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:15.166954 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mfnxk" event={"ID":"76a5a648-0585-408d-8556-c0e31dcea942","Type":"ContainerStarted","Data":"29a1c93d9ff25a9c2e13926a8064e953e59521f7ba9cb4a8f1d942db6346fdde"} Nov 01 22:22:15.180804 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:15.180762 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" podStartSLOduration=3.199474665 podStartE2EDuration="11.180748038s" totalImagesPullingTime="7.981273373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:15.180078454 +0000 UTC m=+11.707006222" watchObservedRunningTime="2026-11-01 22:22:15.180748038 +0000 UTC m=+11.707675800" Nov 01 22:22:15.217467 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:15.217418 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-mfnxk" podStartSLOduration=3.114577081 podStartE2EDuration="11.217403187s" totalImagesPullingTime="8.102826106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:15.217020126 +0000 UTC m=+11.743947894" watchObservedRunningTime="2026-11-01 22:22:15.217403187 +0000 UTC m=+11.744330952" Nov 01 22:22:15.375203 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:15.375161 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:15.375606 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:15.375585 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:15.994574 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:15.994539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:15.994759 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:15.994702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:15.994759 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:15.994745 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:15.994878 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:15.994832 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:16.170819 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:16.170778 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v2bl8" event={"ID":"e1dd1e50-ecd0-4724-862d-752e9e1aecd0","Type":"ContainerStarted","Data":"9d653afa127cfa85bfa27a24b96bfdb4823de964edde0e2509156695eef7f231"} Nov 01 22:22:16.184752 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:16.184688 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v2bl8" podStartSLOduration=4.423516825 podStartE2EDuration="12.184675156s" totalImagesPullingTime="7.761158331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:16.184292304 +0000 UTC m=+12.711220071" watchObservedRunningTime="2026-11-01 22:22:16.184675156 +0000 UTC m=+12.711602922" Nov 01 22:22:16.994415 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:16.994384 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:16.994602 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:16.994498 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:17.174880 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:17.174847 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="340dc9ed9923b69c7bb9303140fced897a53be352d54f07c1dd57eb200cff9a7" exitCode=0 Nov 01 22:22:17.175492 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:17.174920 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" event={"ID":"4d408013-e118-42d1-809f-5d3b50f12565","Type":"ContainerDied","Data":"340dc9ed9923b69c7bb9303140fced897a53be352d54f07c1dd57eb200cff9a7"} Nov 01 22:22:17.176641 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:17.176615 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" event={"ID":"f2c4e933-b200-47a6-94c8-1966aeb16884","Type":"ContainerStarted","Data":"ece2aa54aee96d81f4d3f4b15717a15cbd6e9ccab4fc4db11de7f6fd44f9d02a"} Nov 01 22:22:17.210203 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:17.210144 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" podStartSLOduration=3.122982837 podStartE2EDuration="13.210129857s" totalImagesPullingTime="10.08714702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:17.209702019 +0000 UTC m=+13.736629784" watchObservedRunningTime="2026-11-01 22:22:17.210129857 +0000 UTC m=+13.737057631" Nov 01 22:22:17.995351 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:17.995155 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:17.995508 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:17.995234 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:17.995575 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:17.995503 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:17.995575 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:17.995550 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:18.995302 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:18.995259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:18.995764 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:18.995398 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:19.182306 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:19.182272 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="6b718c65ea4aec999d784c5c7d5a7ea645e6c6088b7ecbac5caa2dc00086212c" exitCode=0 Nov 01 22:22:19.182306 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:19.182310 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" event={"ID":"4d408013-e118-42d1-809f-5d3b50f12565","Type":"ContainerDied","Data":"6b718c65ea4aec999d784c5c7d5a7ea645e6c6088b7ecbac5caa2dc00086212c"} Nov 01 22:22:19.996443 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:19.996411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:19.996871 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:19.996536 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:19.996933 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:19.996909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:19.997005 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:19.996988 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:20.994759 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:20.994715 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:20.994997 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:20.994833 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:21.592858 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:21.592774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:21.592858 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:21.592822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:21.593387 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:21.592948 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:21.593387 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:21.592956 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:21.593387 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:21.593017 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs podName:b21a7a1c-6db6-422d-a24f-5bdc6b4a700c nodeName:}" failed. No retries permitted until 2026-11-01 22:22:37.592997044 +0000 UTC m=+34.119924812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs") pod "network-metrics-daemon-8cbk8" (UID: "b21a7a1c-6db6-422d-a24f-5bdc6b4a700c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:21.593387 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:21.593035 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret podName:8739003e-fba1-4d47-80e9-0f484d117b73 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:37.593026529 +0000 UTC m=+34.119954277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret") pod "global-pull-secret-syncer-cg4rv" (UID: "8739003e-fba1-4d47-80e9-0f484d117b73") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:21.694162 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:21.694092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:21.694336 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:21.694271 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:21.694336 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:21.694296 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:21.694336 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:21.694306 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7w5t7 for pod openshift-network-diagnostics/network-check-target-vzk9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:21.694500 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:21.694357 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7 podName:4dbe2700-733d-4ebd-9f47-7b9f30bb62fd nodeName:}" failed. No retries permitted until 2026-11-01 22:22:37.694344866 +0000 UTC m=+34.221272623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w5t7" (UniqueName: "kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7") pod "network-check-target-vzk9n" (UID: "4dbe2700-733d-4ebd-9f47-7b9f30bb62fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:21.995459 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:21.995377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:21.995459 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:21.995416 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:21.995658 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:21.995533 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:21.995723 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:21.995646 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:22.192074 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:22.192036 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="ff32440f3b9dc4bb22abeea5de0926a86257971dca235095624a8a96e7c791ad" exitCode=0 Nov 01 22:22:22.192244 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:22.192101 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" event={"ID":"4d408013-e118-42d1-809f-5d3b50f12565","Type":"ContainerDied","Data":"ff32440f3b9dc4bb22abeea5de0926a86257971dca235095624a8a96e7c791ad"} Nov 01 22:22:22.995127 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:22.995075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:22.995602 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:22.995230 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:23.197980 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.197943 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="4e8966d105cd3ab3ce28405fca93644ce16b9675f66cc6b20c8cb5f55d431e46" exitCode=0 Nov 01 22:22:23.197980 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.197971 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" event={"ID":"4d408013-e118-42d1-809f-5d3b50f12565","Type":"ContainerDied","Data":"4e8966d105cd3ab3ce28405fca93644ce16b9675f66cc6b20c8cb5f55d431e46"} Nov 01 22:22:23.259057 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.259021 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxlz9_e1baadc8-f14c-490f-9655-a55352e7eb8b/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:23.419619 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.419587 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxlz9_e1baadc8-f14c-490f-9655-a55352e7eb8b/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:23.598722 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.598642 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxlz9_e1baadc8-f14c-490f-9655-a55352e7eb8b/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:23.692511 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.692487 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:23.692638 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.692624 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:23.693041 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.693021 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:22:23.734570 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.734545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxlz9_e1baadc8-f14c-490f-9655-a55352e7eb8b/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:23.852713 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.852473 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fxlz9_e1baadc8-f14c-490f-9655-a55352e7eb8b/konnectivity-agent/0.log" limit=1024 Nov 01 22:22:23.995014 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.994988 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:23.995155 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:23.995078 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:23.995624 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:23.995152 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:23.995624 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:23.995278 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:24.202898 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:24.202812 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" event={"ID":"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c","Type":"ContainerStarted","Data":"f02edf4c048eaa9ecd772ada6a9e267ef8a87a94dc2f45b0581d9a8e77b148ae"} Nov 01 22:22:24.202898 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:24.202897 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" event={"ID":"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c","Type":"ContainerStarted","Data":"49ae10239b36821cd076fa7c34c24e7720953e6a0896abb2ba881959b00207b0"} Nov 01 22:22:24.203086 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:24.202937 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" event={"ID":"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c","Type":"ContainerStarted","Data":"cceef103b7f9fa0a8c78c8eb63ba11ed02391d6bf7185a69650b00087a32ecf1"} Nov 01 22:22:24.203086 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:24.202972 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" event={"ID":"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c","Type":"ContainerStarted","Data":"086a5e7eec9e51630dd85dd267fad8ca5a4cdb99d52796feb2b559913f90752d"} Nov 01 22:22:24.203086 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:24.203015 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" event={"ID":"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c","Type":"ContainerStarted","Data":"a5fe697e248a8eb3acbbcdeadf1fa0d4017a7d27eb71d63a62d5590672bed7b6"} Nov 01 22:22:24.203086 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:24.203047 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" event={"ID":"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c","Type":"ContainerStarted","Data":"25ac60b5c8c3afa08b422e61a0b70bc06ce32038ed0db5c26f31856f15ccea7f"} Nov 01 22:22:24.205928 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:24.205902 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" event={"ID":"4d408013-e118-42d1-809f-5d3b50f12565","Type":"ContainerStarted","Data":"4d0307d3eee808e8e4e464229e8f5a1c167e16ad7c84700d6990b680cc5b806f"} Nov 01 22:22:24.226163 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:24.226096 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t8rz8" podStartSLOduration=8.909225159 podStartE2EDuration="20.22608491s" totalImagesPullingTime="11.316859751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:24.224921114 +0000 UTC m=+20.751848880" watchObservedRunningTime="2026-11-01 22:22:24.22608491 +0000 UTC m=+20.753012676" Nov 01 22:22:24.428511 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:24.428486 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mfnxk_76a5a648-0585-408d-8556-c0e31dcea942/dns-node-resolver/0.log" Nov 01 22:22:24.995227 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:24.995197 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:24.995656 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:24.995306 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:25.027949 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:25.027922 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f9snb_6962a1b2-43de-4da8-a1f4-af8101d42fd0/node-ca/0.log" Nov 01 22:22:25.994879 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:25.994856 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:25.994971 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:25.994899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:25.994971 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:25.994949 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:25.995069 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:25.995020 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:26.213950 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:26.213920 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" event={"ID":"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c","Type":"ContainerStarted","Data":"6b4ea72081793bd1befac9eee8f198c6024f3199909202a0f6923404c7fdabd8"} Nov 01 22:22:26.994893 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:26.994854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:26.995095 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:26.994971 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:27.995250 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:27.995212 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:27.995700 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:27.995255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:27.995700 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:27.995360 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:27.995700 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:27.995468 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:28.221905 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:28.221872 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" event={"ID":"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c","Type":"ContainerStarted","Data":"eb3a19c9afeb7e9374264c70db3ce4328b1f686b29685a3842517242640766f4"} Nov 01 22:22:28.222159 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:28.222140 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:28.237454 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:28.237286 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:28.247899 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:28.247817 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" podStartSLOduration=6.834158414 podStartE2EDuration="24.247805026s" totalImagesPullingTime="17.413646612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:28.247559633 +0000 UTC m=+24.774487398" watchObservedRunningTime="2026-11-01 22:22:28.247805026 +0000 UTC m=+24.774732791" Nov 01 22:22:28.994454 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:28.994421 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:28.994606 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:28.994522 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:29.224371 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:29.224340 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:29.225099 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:29.224844 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:29.240121 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:29.240089 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:29.994506 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:29.994475 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:29.994506 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:29.994506 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:29.994718 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:29.994594 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:29.994718 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:29.994698 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:30.226600 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:30.226573 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:30.995053 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:30.995020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:30.995243 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:30.995124 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:31.228881 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:31.228856 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:31.994733 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:31.994701 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:31.994907 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:31.994823 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:31.994907 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:31.994873 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:31.994981 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:31.994962 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:32.420967 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:32.420938 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:22:32.994775 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:32.994746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:32.994948 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:32.994837 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:33.995583 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:33.995552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:33.995996 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:33.995642 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:33.995996 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:33.995731 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:33.995996 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:33.995839 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:34.994735 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:34.994696 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:34.994906 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:34.994802 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:35.994922 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:35.994892 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:35.995366 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:35.994996 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:35.995366 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:35.995100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:35.995366 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:35.995288 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:36.994885 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:36.994852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:36.995058 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:36.994952 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:37.603689 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:37.603646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:37.603689 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:37.603688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:37.603904 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:37.603794 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:37.603904 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:37.603796 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:37.603904 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:37.603843 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs podName:b21a7a1c-6db6-422d-a24f-5bdc6b4a700c nodeName:}" failed. No retries permitted until 2026-11-01 22:23:09.603830347 +0000 UTC m=+66.130758091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs") pod "network-metrics-daemon-8cbk8" (UID: "b21a7a1c-6db6-422d-a24f-5bdc6b4a700c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:37.603904 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:37.603856 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret podName:8739003e-fba1-4d47-80e9-0f484d117b73 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:09.60384985 +0000 UTC m=+66.130777594 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret") pod "global-pull-secret-syncer-cg4rv" (UID: "8739003e-fba1-4d47-80e9-0f484d117b73") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:37.704369 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:37.704333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:37.704529 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:37.704452 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:37.704529 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:37.704468 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:37.704529 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:37.704477 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7w5t7 for pod openshift-network-diagnostics/network-check-target-vzk9n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:37.704529 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:37.704523 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7 podName:4dbe2700-733d-4ebd-9f47-7b9f30bb62fd nodeName:}" failed. No retries permitted until 2026-11-01 22:23:09.704510627 +0000 UTC m=+66.231438372 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w5t7" (UniqueName: "kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7") pod "network-check-target-vzk9n" (UID: "4dbe2700-733d-4ebd-9f47-7b9f30bb62fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:37.999952 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:37.999880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:37.999952 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:37.999911 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:38.000375 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:37.999980 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:38.000375 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:38.000034 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:38.994352 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:38.994321 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:38.994511 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:38.994450 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:39.056365 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:39.056278 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cg4rv"] Nov 01 22:22:39.056809 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:39.056429 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:39.056809 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:39.056555 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:39.065521 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:39.065493 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzk9n"] Nov 01 22:22:39.065641 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:39.065560 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8cbk8"] Nov 01 22:22:39.065641 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:39.065637 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:39.065779 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:39.065751 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:39.245202 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:39.245177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:39.245355 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:39.245283 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:40.143556 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:40.143531 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mfnxk_76a5a648-0585-408d-8556-c0e31dcea942/dns-node-resolver/0.log" Nov 01 22:22:40.942890 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:40.942859 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f9snb_6962a1b2-43de-4da8-a1f4-af8101d42fd0/node-ca/0.log" Nov 01 22:22:40.995305 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:40.995274 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:40.995305 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:40.995301 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:40.995472 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:40.995312 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:40.995472 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:40.995407 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:40.995544 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:40.995497 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:40.995584 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:40.995555 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:42.994897 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:42.994858 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:42.995363 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:42.994925 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:42.995363 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:42.994950 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:42.995363 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:42.995030 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vzk9n" podUID="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" Nov 01 22:22:42.995363 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:42.995097 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cg4rv" podUID="8739003e-fba1-4d47-80e9-0f484d117b73" Nov 01 22:22:42.995363 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:42.995189 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cbk8" podUID="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" Nov 01 22:22:43.758650 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.758490 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeReady" Nov 01 22:22:43.758810 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.758787 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 22:22:43.801375 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.801341 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kc74w"] Nov 01 22:22:43.823675 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.823645 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-77bdk"] Nov 01 22:22:43.823816 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.823751 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:43.828625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.828598 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 22:22:43.828625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.828617 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 22:22:43.828796 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.828632 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 22:22:43.828796 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.828700 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4bt99\"" Nov 01 22:22:43.828796 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.828771 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 22:22:43.833598 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.833582 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kc74w"] Nov 01 22:22:43.833682 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.833617 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-77bdk"] Nov 01 22:22:43.833718 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.833688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:22:43.838456 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.838439 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 22:22:43.838456 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.838441 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 22:22:43.838595 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.838495 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 22:22:43.838595 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.838548 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4jwn7\"" Nov 01 22:22:43.902558 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.902531 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sh7c7"] Nov 01 22:22:43.923928 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.923902 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sh7c7"] Nov 01 22:22:43.924051 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.924020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:43.928155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.928132 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 22:22:43.928155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.928135 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4kglz\"" Nov 01 22:22:43.928331 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.928135 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 22:22:43.955675 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.955646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fhtq\" (UniqueName: \"kubernetes.io/projected/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-kube-api-access-6fhtq\") pod \"ingress-canary-77bdk\" (UID: \"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d\") " pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:22:43.955675 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.955674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-data-volume\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:43.955835 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.955694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-cert\") pod \"ingress-canary-77bdk\" (UID: \"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d\") " pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:22:43.955835 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.955714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-crio-socket\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:43.955835 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.955820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5csc\" (UniqueName: \"kubernetes.io/projected/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-api-access-h5csc\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:43.955921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.955845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:43.955921 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:43.955863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.056724 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.056635 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26366d8c-76f1-479e-9715-6512f20bd3c6-tmp-dir\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.056724 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.056684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-cert\") pod \"ingress-canary-77bdk\" (UID: \"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d\") " pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:22:44.056724 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.056713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-crio-socket\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.057261 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.056744 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26366d8c-76f1-479e-9715-6512f20bd3c6-config-volume\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.057261 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.056867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctgxg\" (UniqueName: \"kubernetes.io/projected/26366d8c-76f1-479e-9715-6512f20bd3c6-kube-api-access-ctgxg\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.057261 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.056921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5csc\" (UniqueName: \"kubernetes.io/projected/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-api-access-h5csc\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.057261 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.056933 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-crio-socket\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.057261 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.056984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.057261 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.057016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.057261 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.057081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26366d8c-76f1-479e-9715-6512f20bd3c6-metrics-tls\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.057261 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.057124 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fhtq\" (UniqueName: \"kubernetes.io/projected/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-kube-api-access-6fhtq\") pod \"ingress-canary-77bdk\" (UID: \"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d\") " pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:22:44.057261 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.057176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-data-volume\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.057600 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.057448 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-data-volume\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.057600 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.057568 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.060776 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.060747 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.060882 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.060808 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-cert\") pod \"ingress-canary-77bdk\" (UID: \"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d\") " pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:22:44.065768 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.065741 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5csc\" (UniqueName: \"kubernetes.io/projected/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-api-access-h5csc\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.066148 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.066125 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fhtq\" (UniqueName: \"kubernetes.io/projected/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-kube-api-access-6fhtq\") pod \"ingress-canary-77bdk\" (UID: \"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d\") " pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:22:44.157938 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.157902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctgxg\" (UniqueName: \"kubernetes.io/projected/26366d8c-76f1-479e-9715-6512f20bd3c6-kube-api-access-ctgxg\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.158145 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.157983 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:22:44.158145 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.158077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26366d8c-76f1-479e-9715-6512f20bd3c6-metrics-tls\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.158145 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.158123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26366d8c-76f1-479e-9715-6512f20bd3c6-tmp-dir\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.158279 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.158158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26366d8c-76f1-479e-9715-6512f20bd3c6-config-volume\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.158498 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.158478 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26366d8c-76f1-479e-9715-6512f20bd3c6-tmp-dir\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.158597 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.158580 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26366d8c-76f1-479e-9715-6512f20bd3c6-config-volume\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.160176 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.160157 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26366d8c-76f1-479e-9715-6512f20bd3c6-metrics-tls\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.165917 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.165896 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctgxg\" (UniqueName: \"kubernetes.io/projected/26366d8c-76f1-479e-9715-6512f20bd3c6-kube-api-access-ctgxg\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.165996 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.165975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:22:44.264965 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.264931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:44.498214 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.498180 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-77bdk"] Nov 01 22:22:44.498732 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:44.498707 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bfd9f9_84c5_48b4_bbd1_d1f0611a8b2d.slice/crio-1aea23dcf3e0b02275ede97637ab172be1c6bfd2b9618d3aac9b2bc5f11acaa4 WatchSource:0}: Error finding container 1aea23dcf3e0b02275ede97637ab172be1c6bfd2b9618d3aac9b2bc5f11acaa4: Status 404 returned error can't find the container with id 1aea23dcf3e0b02275ede97637ab172be1c6bfd2b9618d3aac9b2bc5f11acaa4 Nov 01 22:22:44.499910 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.499888 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kc74w"] Nov 01 22:22:44.501545 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:44.501518 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc2d08c_c26b_41b1_a36d_76dc6ab9fcf0.slice/crio-9ff711848f297a3eba2d1e200eb8c14f77a2569882c909d8515bb893f7830428 WatchSource:0}: Error finding container 9ff711848f297a3eba2d1e200eb8c14f77a2569882c909d8515bb893f7830428: Status 404 returned error can't find the container with id 9ff711848f297a3eba2d1e200eb8c14f77a2569882c909d8515bb893f7830428 Nov 01 22:22:44.548362 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.548222 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sh7c7"] Nov 01 22:22:44.551077 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:44.551054 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26366d8c_76f1_479e_9715_6512f20bd3c6.slice/crio-74dba1cf8098e916cca03e7c5c84cae0255d622afb2a200ada024e5531890363 WatchSource:0}: Error finding container 74dba1cf8098e916cca03e7c5c84cae0255d622afb2a200ada024e5531890363: Status 404 returned error can't find the container with id 74dba1cf8098e916cca03e7c5c84cae0255d622afb2a200ada024e5531890363 Nov 01 22:22:44.994521 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.994491 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:22:44.994521 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.994528 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:22:44.994764 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:44.994667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:22:45.000371 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:45.000328 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fpxpg\"" Nov 01 22:22:45.000371 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:45.000331 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:22:45.000371 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:45.000345 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:22:45.000371 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:45.000375 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5scqq\"" Nov 01 22:22:45.000576 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:45.000421 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:22:45.000576 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:45.000431 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9pl6q\"" Nov 01 22:22:45.000576 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:45.000483 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:22:45.259930 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:45.259832 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc74w" event={"ID":"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0","Type":"ContainerStarted","Data":"03866be3db68fe8ed672b9e106284ad194557781a14955208cef25f41ea3bab9"} Nov 01 22:22:45.259930 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:45.259915 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc74w" event={"ID":"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0","Type":"ContainerStarted","Data":"9ff711848f297a3eba2d1e200eb8c14f77a2569882c909d8515bb893f7830428"} Nov 01 22:22:45.261475 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:45.261442 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sh7c7" event={"ID":"26366d8c-76f1-479e-9715-6512f20bd3c6","Type":"ContainerStarted","Data":"74dba1cf8098e916cca03e7c5c84cae0255d622afb2a200ada024e5531890363"} Nov 01 22:22:45.262820 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:45.262796 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-77bdk" event={"ID":"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d","Type":"ContainerStarted","Data":"1aea23dcf3e0b02275ede97637ab172be1c6bfd2b9618d3aac9b2bc5f11acaa4"} Nov 01 22:22:46.267005 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:46.266977 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc74w" event={"ID":"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0","Type":"ContainerStarted","Data":"adbb1df9949fc787984f0341f58ec7a42b220132158fcea1797b87480e4d9639"} Nov 01 22:22:46.939407 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:46.939366 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cw7qc"] Nov 01 22:22:46.967781 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:46.967745 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz"] Nov 01 22:22:46.968876 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:46.968851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:46.978034 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:46.978008 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv"] Nov 01 22:22:46.978186 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:46.978169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:46.984461 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:46.984433 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz"] Nov 01 22:22:46.984558 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:46.984484 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv"] Nov 01 22:22:46.984626 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:46.984560 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.015625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.015599 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 22:22:47.015759 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.015687 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 22:22:47.015759 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.015715 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 22:22:47.015970 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.015955 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-t7pjg\"" Nov 01 22:22:47.016122 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.016090 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 22:22:47.016216 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.016207 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 22:22:47.016510 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.016493 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zj5nt\"" Nov 01 22:22:47.016584 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.016544 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 22:22:47.016584 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.016551 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-n7fmc\"" Nov 01 22:22:47.033796 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.033769 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 22:22:47.034132 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.034082 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 22:22:47.034229 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.034096 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 22:22:47.034229 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.034192 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 22:22:47.034229 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.034137 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 22:22:47.084980 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.084946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-root\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.085178 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085001 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c577v\" (UniqueName: \"kubernetes.io/projected/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-kube-api-access-c577v\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.085178 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2r9f\" (UniqueName: \"kubernetes.io/projected/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-api-access-c2r9f\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.085178 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085058 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bntx6\" (UniqueName: \"kubernetes.io/projected/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-kube-api-access-bntx6\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.085178 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085160 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e9ed2868-24ad-473c-bac4-01344c0e4bed-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.085385 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.085385 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.085385 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085291 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.085385 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085318 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-metrics-client-ca\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.085385 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-accelerators-collector-config\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.085625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085387 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.085625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-sys\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.085625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-tls\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.085625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.085625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-textfile\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.085625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-wtmp\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.085625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.085625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.085625 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.085619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.186643 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.186854 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.186854 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-metrics-client-ca\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.186854 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-accelerators-collector-config\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.186854 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.186854 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-sys\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.186854 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-tls\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.186854 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.187246 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-textfile\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.187246 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-wtmp\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.187246 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.187246 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.187246 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.186988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.187246 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.187015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-root\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.187246 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.187050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c577v\" (UniqueName: \"kubernetes.io/projected/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-kube-api-access-c577v\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.187246 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.187076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2r9f\" (UniqueName: \"kubernetes.io/projected/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-api-access-c2r9f\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.187246 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.187121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntx6\" (UniqueName: \"kubernetes.io/projected/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-kube-api-access-bntx6\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.187246 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.187151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e9ed2868-24ad-473c-bac4-01344c0e4bed-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.187246 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.187187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.187899 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.187873 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-sys\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.188099 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.188078 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-textfile\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.188486 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.188461 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-accelerators-collector-config\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.188594 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.188565 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.189245 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.189219 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.189351 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.189227 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.189562 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.189491 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-root\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.189644 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.189569 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-wtmp\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.189740 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.189707 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-metrics-client-ca\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.190894 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.190828 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e9ed2868-24ad-473c-bac4-01344c0e4bed-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.191307 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.191283 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.192169 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.192145 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.192597 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.192554 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.194155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.193861 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.194268 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.194246 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.199942 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.199881 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bntx6\" (UniqueName: \"kubernetes.io/projected/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-kube-api-access-bntx6\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.200932 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.200907 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c577v\" (UniqueName: \"kubernetes.io/projected/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-kube-api-access-c577v\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.202690 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.202665 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-tls\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.203010 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.202988 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2r9f\" (UniqueName: \"kubernetes.io/projected/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-api-access-c2r9f\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.310467 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.310433 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:22:47.352277 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.352240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:22:47.360412 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.360380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:22:47.374468 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:47.374425 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77255dd_93d5_4c30_8a4c_f4c7ea44d702.slice/crio-a861487740395d771feca30986d9bd4c21af987fe32b4235c30635cd0547c5b9 WatchSource:0}: Error finding container a861487740395d771feca30986d9bd4c21af987fe32b4235c30635cd0547c5b9: Status 404 returned error can't find the container with id a861487740395d771feca30986d9bd4c21af987fe32b4235c30635cd0547c5b9 Nov 01 22:22:47.658353 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.658318 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz"] Nov 01 22:22:47.667733 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:47.667704 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8572e79_8cf8_4b15_b77f_ba409bd3b51f.slice/crio-dee8af8b6e9db4d10d99cdaf32ee9aa115336aa79d884d90bab5fe69811619ce WatchSource:0}: Error finding container dee8af8b6e9db4d10d99cdaf32ee9aa115336aa79d884d90bab5fe69811619ce: Status 404 returned error can't find the container with id dee8af8b6e9db4d10d99cdaf32ee9aa115336aa79d884d90bab5fe69811619ce Nov 01 22:22:47.678224 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:47.678186 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv"] Nov 01 22:22:47.684159 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:47.684130 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ed2868_24ad_473c_bac4_01344c0e4bed.slice/crio-8c6fe3b2750890436a0c7e02358d55145f299e80ba2ca1f043b8d75ac056398b WatchSource:0}: Error finding container 8c6fe3b2750890436a0c7e02358d55145f299e80ba2ca1f043b8d75ac056398b: Status 404 returned error can't find the container with id 8c6fe3b2750890436a0c7e02358d55145f299e80ba2ca1f043b8d75ac056398b Nov 01 22:22:48.017815 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.015845 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:22:48.028061 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.028041 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.033683 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.033647 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:22:48.036523 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.036501 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 22:22:48.036748 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.036630 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 22:22:48.036816 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.036752 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 22:22:48.036866 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.036844 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 22:22:48.036919 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.036910 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 22:22:48.037039 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.036973 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 22:22:48.037039 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.037037 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 22:22:48.037245 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.037167 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mn4xg\"" Nov 01 22:22:48.037301 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.037255 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 22:22:48.037353 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.037319 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 22:22:48.097577 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-web-config\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.097733 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78lnx\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-kube-api-access-78lnx\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.097733 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.097856 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.097856 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.097856 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-out\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.097856 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.098050 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.098050 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.098050 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.098050 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.098050 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-volume\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.098050 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.097975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199412 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-web-config\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199412 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78lnx\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-kube-api-access-78lnx\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199596 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199596 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199596 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199596 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-out\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199596 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199596 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199596 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199826 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199826 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199647 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199826 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-volume\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.199826 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.199701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.201076 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:22:48.200228 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle podName:f12c8a75-38c0-4e45-9ec9-9e625521ceb8 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:48.700205163 +0000 UTC m=+45.227132927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8") : configmap references non-existent config key: ca-bundle.crt Nov 01 22:22:48.201076 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.200499 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.201427 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.201402 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.204517 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.204492 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.204634 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.204612 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-web-config\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.205630 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.205607 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-out\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.205746 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.205633 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.206464 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.206424 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-volume\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.206819 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.206797 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.207237 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.207216 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.207319 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.207256 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.209234 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.209211 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.211345 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.211323 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78lnx\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-kube-api-access-78lnx\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.276333 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.276295 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cw7qc" event={"ID":"d77255dd-93d5-4c30-8a4c-f4c7ea44d702","Type":"ContainerStarted","Data":"a861487740395d771feca30986d9bd4c21af987fe32b4235c30635cd0547c5b9"} Nov 01 22:22:48.278312 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.278279 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" event={"ID":"e8572e79-8cf8-4b15-b77f-ba409bd3b51f","Type":"ContainerStarted","Data":"94fef7a05abc8fef7dea0b453b972c603274658b0bf7965576f4139661afa208"} Nov 01 22:22:48.278452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.278347 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" event={"ID":"e8572e79-8cf8-4b15-b77f-ba409bd3b51f","Type":"ContainerStarted","Data":"ecf8ad94e9825cfafbe99482769b6c3eb0945120ff6dce080c14bbb1104f6f5f"} Nov 01 22:22:48.278452 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.278378 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" event={"ID":"e8572e79-8cf8-4b15-b77f-ba409bd3b51f","Type":"ContainerStarted","Data":"dee8af8b6e9db4d10d99cdaf32ee9aa115336aa79d884d90bab5fe69811619ce"} Nov 01 22:22:48.280248 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.280217 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sh7c7" event={"ID":"26366d8c-76f1-479e-9715-6512f20bd3c6","Type":"ContainerStarted","Data":"548811da0fd33d17bc84b171357caf59c801a4c2507fe8a48c558444a5c9f1f7"} Nov 01 22:22:48.280346 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.280267 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sh7c7" event={"ID":"26366d8c-76f1-479e-9715-6512f20bd3c6","Type":"ContainerStarted","Data":"4d5a7f160706acd88e1d81451b748de072dd2c05d8222aba0c0a4264c9b12bc4"} Nov 01 22:22:48.280346 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.280326 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sh7c7" Nov 01 22:22:48.282280 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.282254 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kc74w" event={"ID":"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0","Type":"ContainerStarted","Data":"0ba85a9e74e65e1774e99a867bb4f551272fb3bf316a33e937b90661960b2eb4"} Nov 01 22:22:48.283421 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.283396 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" event={"ID":"e9ed2868-24ad-473c-bac4-01344c0e4bed","Type":"ContainerStarted","Data":"8c6fe3b2750890436a0c7e02358d55145f299e80ba2ca1f043b8d75ac056398b"} Nov 01 22:22:48.319100 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.319042 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-sh7c7" podStartSLOduration=2.184640033 podStartE2EDuration="5.319024886s" totalImagesPullingTime="3.134384853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:48.298988031 +0000 UTC m=+44.825915801" watchObservedRunningTime="2026-11-01 22:22:48.319024886 +0000 UTC m=+44.845952654" Nov 01 22:22:48.320001 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.319772 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kc74w" podStartSLOduration=2.100288759 podStartE2EDuration="5.319761418s" totalImagesPullingTime="3.219472659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:48.318754493 +0000 UTC m=+44.845682259" watchObservedRunningTime="2026-11-01 22:22:48.319761418 +0000 UTC m=+44.846689223" Nov 01 22:22:48.704448 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.704417 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.705293 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.705233 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:48.985604 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:48.985520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:22:49.060455 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.060423 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-675964fd8b-v45n9"] Nov 01 22:22:49.063835 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.063809 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.068939 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.068911 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 22:22:49.069129 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.068940 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 22:22:49.069129 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.069004 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 22:22:49.069129 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.069026 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 22:22:49.069288 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.069137 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 22:22:49.069288 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.069145 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7b27s7rg7h8jo\"" Nov 01 22:22:49.069288 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.069252 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-w9c7x\"" Nov 01 22:22:49.074038 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.074015 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-675964fd8b-v45n9"] Nov 01 22:22:49.107653 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.107616 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.107653 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.107666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-metrics-client-ca\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.107872 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.107802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-tmp\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.107927 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.107866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.107927 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.107906 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-tls\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.108018 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.107977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.108018 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.108005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.108129 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.108036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-grpc-tls\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.108187 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.108120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44ld4\" (UniqueName: \"kubernetes.io/projected/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-kube-api-access-44ld4\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.208880 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.208853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.209016 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.208888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.209016 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.208908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-grpc-tls\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.209016 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.208924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ld4\" (UniqueName: \"kubernetes.io/projected/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-kube-api-access-44ld4\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.209016 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.208962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.209016 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.208989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-metrics-client-ca\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.209260 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.209045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-tmp\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.209260 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.209100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.209260 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.209141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-tls\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.210033 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.209829 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-tmp\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.210342 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.210317 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-metrics-client-ca\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.213345 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.213211 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.213345 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.213276 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.213345 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.213277 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.213345 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.213318 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.213599 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.213517 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-tls\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.213869 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.213850 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-grpc-tls\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.216599 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.216535 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ld4\" (UniqueName: \"kubernetes.io/projected/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-kube-api-access-44ld4\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.273923 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.273845 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:22:49.277262 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:49.277233 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12c8a75_38c0_4e45_9ec9_9e625521ceb8.slice/crio-46844a8e32c7b4cfa6f04a52ced4f17086e0311f0f1d2fa694589b7525f975b8 WatchSource:0}: Error finding container 46844a8e32c7b4cfa6f04a52ced4f17086e0311f0f1d2fa694589b7525f975b8: Status 404 returned error can't find the container with id 46844a8e32c7b4cfa6f04a52ced4f17086e0311f0f1d2fa694589b7525f975b8 Nov 01 22:22:49.289450 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.289423 2580 generic.go:356] "Generic (PLEG): container finished" podID="d77255dd-93d5-4c30-8a4c-f4c7ea44d702" containerID="8c95e5f56b5cc01888ef445435aa21a5ba1acaa7343cb3eae07fb88686439a45" exitCode=0 Nov 01 22:22:49.289553 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.289491 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cw7qc" event={"ID":"d77255dd-93d5-4c30-8a4c-f4c7ea44d702","Type":"ContainerDied","Data":"8c95e5f56b5cc01888ef445435aa21a5ba1acaa7343cb3eae07fb88686439a45"} Nov 01 22:22:49.290864 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.290838 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerStarted","Data":"46844a8e32c7b4cfa6f04a52ced4f17086e0311f0f1d2fa694589b7525f975b8"} Nov 01 22:22:49.293966 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.293851 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" event={"ID":"e8572e79-8cf8-4b15-b77f-ba409bd3b51f","Type":"ContainerStarted","Data":"06e0984ba9cbe51d44fee798557cbaa0c27780eef85af0f4ebe029e5bb6db33f"} Nov 01 22:22:49.296269 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.295986 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-77bdk" event={"ID":"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d","Type":"ContainerStarted","Data":"99cfe17226cf39a7b3e41d3f35955d0ae0d37042e0d8a6a5afdf45803434abb4"} Nov 01 22:22:49.331386 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.331173 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-77bdk" podStartSLOduration=2.491238907 podStartE2EDuration="6.331154043s" totalImagesPullingTime="3.839915136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:49.329952522 +0000 UTC m=+45.856880288" watchObservedRunningTime="2026-11-01 22:22:49.331154043 +0000 UTC m=+45.858081811" Nov 01 22:22:49.348120 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.348065 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" podStartSLOduration=2.156446497 podStartE2EDuration="3.348051848s" totalImagesPullingTime="1.191605351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:49.347611393 +0000 UTC m=+45.874539162" watchObservedRunningTime="2026-11-01 22:22:49.348051848 +0000 UTC m=+45.874979614" Nov 01 22:22:49.382741 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.382715 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:49.658506 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:49.658470 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-675964fd8b-v45n9"] Nov 01 22:22:49.661630 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:49.661602 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd5ecdb_5bcd_44ea_8b20_07bdfe8e2f55.slice/crio-1629c22a269b002d6230d7524a8ff25fa938d89393541ddd3513edd6ea73953a WatchSource:0}: Error finding container 1629c22a269b002d6230d7524a8ff25fa938d89393541ddd3513edd6ea73953a: Status 404 returned error can't find the container with id 1629c22a269b002d6230d7524a8ff25fa938d89393541ddd3513edd6ea73953a Nov 01 22:22:50.305140 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:50.303999 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cw7qc" event={"ID":"d77255dd-93d5-4c30-8a4c-f4c7ea44d702","Type":"ContainerStarted","Data":"1ce83a3e86328dac8e1ba8f8b65cf7e0a7e4860fee1501d88cfb5992652d7407"} Nov 01 22:22:50.305140 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:50.304050 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cw7qc" event={"ID":"d77255dd-93d5-4c30-8a4c-f4c7ea44d702","Type":"ContainerStarted","Data":"68f13cecf24e0cdfabccb3352a70342fb666a4629c4146828c0da2877679d6fc"} Nov 01 22:22:50.307408 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:50.307239 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" event={"ID":"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55","Type":"ContainerStarted","Data":"1629c22a269b002d6230d7524a8ff25fa938d89393541ddd3513edd6ea73953a"} Nov 01 22:22:50.311296 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:50.309765 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" event={"ID":"e9ed2868-24ad-473c-bac4-01344c0e4bed","Type":"ContainerStarted","Data":"4f6600933ef3814940e29c500310988206c912da0bb88fbab0ca5c3a8f5bec23"} Nov 01 22:22:50.327005 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:50.326080 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cw7qc" podStartSLOduration=2.935211399 podStartE2EDuration="4.326060074s" totalImagesPullingTime="1.314816981s" totalInitContainerRuntime="76.031694ms" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:50.325127959 +0000 UTC m=+46.852055723" watchObservedRunningTime="2026-11-01 22:22:50.326060074 +0000 UTC m=+46.852987841" Nov 01 22:22:51.313935 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.313895 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" event={"ID":"e9ed2868-24ad-473c-bac4-01344c0e4bed","Type":"ContainerStarted","Data":"42b1024b7394c7368d41f052d4fa73924dd7a9f71bb16975c275312dd13d457a"} Nov 01 22:22:51.314821 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.313966 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" event={"ID":"e9ed2868-24ad-473c-bac4-01344c0e4bed","Type":"ContainerStarted","Data":"5c1bba085da4b3a02a75812fca948f1d20fe51877bd0f148d83aca91cf13a2aa"} Nov 01 22:22:51.316161 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.316132 2580 generic.go:356] "Generic (PLEG): container finished" podID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerID="d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8" exitCode=0 Nov 01 22:22:51.316284 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.316220 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerDied","Data":"d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8"} Nov 01 22:22:51.336645 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.336604 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" podStartSLOduration=2.839374304 podStartE2EDuration="5.336593645s" totalImagesPullingTime="2.497219341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:51.335437642 +0000 UTC m=+47.862365443" watchObservedRunningTime="2026-11-01 22:22:51.336593645 +0000 UTC m=+47.863521410" Nov 01 22:22:51.340063 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.340039 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-796bf69784-66sjq"] Nov 01 22:22:51.343710 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.343690 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.355275 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.355251 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-796bf69784-66sjq"] Nov 01 22:22:51.355477 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.355459 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 22:22:51.355614 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.355600 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 22:22:51.388783 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.388753 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 22:22:51.389177 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.388865 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 22:22:51.389177 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.388979 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c91l7l24aggst\"" Nov 01 22:22:51.389177 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.389052 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8x5xq\"" Nov 01 22:22:51.428022 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.427989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/18595f28-dc19-4df0-a61b-905e22653608-audit-log\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.428245 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.428215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.428426 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.428337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-client-certs\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.428426 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.428369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-metrics-server-audit-profiles\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.428683 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.428658 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9zh\" (UniqueName: \"kubernetes.io/projected/18595f28-dc19-4df0-a61b-905e22653608-kube-api-access-lm9zh\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.428834 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.428816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-tls\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.428916 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.428890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-client-ca-bundle\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.529340 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.529302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-tls\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.529510 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.529356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-client-ca-bundle\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.529510 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.529413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/18595f28-dc19-4df0-a61b-905e22653608-audit-log\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.529510 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.529442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.529510 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.529471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-client-certs\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.529510 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.529493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-metrics-server-audit-profiles\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.529745 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.529541 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9zh\" (UniqueName: \"kubernetes.io/projected/18595f28-dc19-4df0-a61b-905e22653608-kube-api-access-lm9zh\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.530442 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.530387 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.530577 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.530505 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/18595f28-dc19-4df0-a61b-905e22653608-audit-log\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.531051 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.531023 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-metrics-server-audit-profiles\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.532146 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.532102 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-tls\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.532223 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.532163 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-client-ca-bundle\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.532223 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.532187 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-client-certs\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.537557 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.537532 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9zh\" (UniqueName: \"kubernetes.io/projected/18595f28-dc19-4df0-a61b-905e22653608-kube-api-access-lm9zh\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.660085 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.660044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:22:51.962021 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:51.961928 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-796bf69784-66sjq"] Nov 01 22:22:51.965313 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:51.965288 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18595f28_dc19_4df0_a61b_905e22653608.slice/crio-1a8667518758b9b4b88869f84365d55fc37b8e0deecc74895a4a13469dda1263 WatchSource:0}: Error finding container 1a8667518758b9b4b88869f84365d55fc37b8e0deecc74895a4a13469dda1263: Status 404 returned error can't find the container with id 1a8667518758b9b4b88869f84365d55fc37b8e0deecc74895a4a13469dda1263 Nov 01 22:22:52.131313 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.131276 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr"] Nov 01 22:22:52.134662 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.134634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.141443 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.141417 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 22:22:52.141606 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.141589 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 22:22:52.141606 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.141598 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 22:22:52.141712 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.141607 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8zdb7\"" Nov 01 22:22:52.141712 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.141640 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 22:22:52.141712 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.141674 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 22:22:52.146296 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.146235 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 22:22:52.148488 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.148466 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr"] Nov 01 22:22:52.235522 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.235443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.235522 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.235497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-federate-client-tls\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.235694 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.235566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-client-tls\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.235694 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.235637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-metrics-client-ca\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.235694 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.235667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgm9\" (UniqueName: \"kubernetes.io/projected/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-kube-api-access-4sgm9\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.235777 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.235729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.235814 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.235783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-serving-certs-ca-bundle\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.235862 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.235849 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.323087 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.323053 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-796bf69784-66sjq" event={"ID":"18595f28-dc19-4df0-a61b-905e22653608","Type":"ContainerStarted","Data":"1a8667518758b9b4b88869f84365d55fc37b8e0deecc74895a4a13469dda1263"} Nov 01 22:22:52.336316 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.336272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-federate-client-tls\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.336493 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.336334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-client-tls\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.336493 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.336365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-metrics-client-ca\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.336493 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.336390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgm9\" (UniqueName: \"kubernetes.io/projected/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-kube-api-access-4sgm9\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.336493 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.336422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.336691 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.336528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-serving-certs-ca-bundle\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.336691 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.336588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.336691 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.336612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.337283 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.337244 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-metrics-client-ca\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.337436 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.337411 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-serving-certs-ca-bundle\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.337822 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.337798 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.339332 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.339304 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-client-tls\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.339420 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.339374 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-federate-client-tls\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.339700 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.339679 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.339912 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.339892 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.343841 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.343822 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgm9\" (UniqueName: \"kubernetes.io/projected/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-kube-api-access-4sgm9\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.454967 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.454934 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:22:52.735818 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:52.735782 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr"] Nov 01 22:22:52.740305 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:22:52.740267 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc034008e_a79f_4ebb_9316_d5d63bc1dbb9.slice/crio-e04aadde8ef47903434fc2edc9676958a2fd296216f41ff779108dc3ab8d7286 WatchSource:0}: Error finding container e04aadde8ef47903434fc2edc9676958a2fd296216f41ff779108dc3ab8d7286: Status 404 returned error can't find the container with id e04aadde8ef47903434fc2edc9676958a2fd296216f41ff779108dc3ab8d7286 Nov 01 22:22:53.331319 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:53.331277 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" event={"ID":"c034008e-a79f-4ebb-9316-d5d63bc1dbb9","Type":"ContainerStarted","Data":"e04aadde8ef47903434fc2edc9676958a2fd296216f41ff779108dc3ab8d7286"} Nov 01 22:22:54.336962 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:54.336935 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" event={"ID":"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55","Type":"ContainerStarted","Data":"bd8aaa2d1f1562c5d5823cf05836ea73b3633adaf7f8477cfc5cd759679f6456"} Nov 01 22:22:54.337364 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:54.337014 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" event={"ID":"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55","Type":"ContainerStarted","Data":"453abe17414d5469ad1950a8be54d6f39ddb8464973acfaf44455824818173a9"} Nov 01 22:22:54.337364 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:54.337061 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" event={"ID":"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55","Type":"ContainerStarted","Data":"d2972e2dce03b65ecc52f15462fe19ed68384e6d964199901accda8725153dfd"} Nov 01 22:22:54.338717 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:54.338690 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-796bf69784-66sjq" event={"ID":"18595f28-dc19-4df0-a61b-905e22653608","Type":"ContainerStarted","Data":"d1ae21f3ca739ffc715e8422c1ccfed7b1464c5fb8f12fc9f850b1da08a966cb"} Nov 01 22:22:55.346529 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:55.346491 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerStarted","Data":"d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7"} Nov 01 22:22:55.348528 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:55.346606 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerStarted","Data":"47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0"} Nov 01 22:22:55.348528 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:55.346655 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerStarted","Data":"bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1"} Nov 01 22:22:55.348528 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:55.346694 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerStarted","Data":"ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab"} Nov 01 22:22:55.348528 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:55.346727 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerStarted","Data":"a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa"} Nov 01 22:22:55.349441 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:55.349414 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" event={"ID":"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55","Type":"ContainerStarted","Data":"8739b92c4e9a1e767c07169ecebae688bc6810a94682256bb937b1c3841c265c"} Nov 01 22:22:55.351901 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:55.351860 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" event={"ID":"c034008e-a79f-4ebb-9316-d5d63bc1dbb9","Type":"ContainerStarted","Data":"1506bb9f68e6888cf379b8fd411c5c15a37dc294e2ae2835222add144936f48d"} Nov 01 22:22:55.351989 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:55.351929 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" event={"ID":"c034008e-a79f-4ebb-9316-d5d63bc1dbb9","Type":"ContainerStarted","Data":"dffddc0e349600aaf0e59992d4d97dd0671133e7ebd3589e8c69e5b802344469"} Nov 01 22:22:55.351989 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:55.351961 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" event={"ID":"c034008e-a79f-4ebb-9316-d5d63bc1dbb9","Type":"ContainerStarted","Data":"ab86bddda50737d88f0ecb3c03c1fff9d87c790cb1d3d95ea134d3d6d4ce9bec"} Nov 01 22:22:55.374806 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:55.374759 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-796bf69784-66sjq" podStartSLOduration=2.620809151 podStartE2EDuration="4.374742278s" totalImagesPullingTime="1.753933127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:54.357094494 +0000 UTC m=+50.884022261" watchObservedRunningTime="2026-11-01 22:22:55.374742278 +0000 UTC m=+51.901670039" Nov 01 22:22:55.376304 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:55.375729 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" podStartSLOduration=1.321343275 podStartE2EDuration="3.375710152s" totalImagesPullingTime="2.054366877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:55.373099248 +0000 UTC m=+51.900027014" watchObservedRunningTime="2026-11-01 22:22:55.375710152 +0000 UTC m=+51.902637919" Nov 01 22:22:56.359812 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:56.359777 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerStarted","Data":"6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653"} Nov 01 22:22:56.367316 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:56.367280 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" event={"ID":"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55","Type":"ContainerStarted","Data":"dd7a5c1181ccbb2633ddffda4165677a1dd3b3836c827a2d7673fd994e1ca983"} Nov 01 22:22:56.367445 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:56.367376 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" event={"ID":"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55","Type":"ContainerStarted","Data":"729d750f539c497db91cedf75a684bc84a1a8bd6069caddf7b6214cf385a1039"} Nov 01 22:22:56.367445 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:56.367429 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:22:56.389762 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:56.389720 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.248727257 podStartE2EDuration="9.389709205s" totalImagesPullingTime="5.126026405s" totalInitContainerRuntime="14.955543ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:56.389184267 +0000 UTC m=+52.916112032" watchObservedRunningTime="2026-11-01 22:22:56.389709205 +0000 UTC m=+52.916636970" Nov 01 22:22:56.414182 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:56.414136 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" podStartSLOduration=2.061621184 podStartE2EDuration="7.41409681s" totalImagesPullingTime="5.352475626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:56.413734691 +0000 UTC m=+52.940662469" watchObservedRunningTime="2026-11-01 22:22:56.41409681 +0000 UTC m=+52.941024576" Nov 01 22:22:58.298222 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:22:58.298186 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sh7c7" Nov 01 22:23:02.377585 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:02.377556 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:02.432249 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:02.432219 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:09.699628 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.699581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:23:09.699628 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.699630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:23:09.703172 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.703153 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:09.703233 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.703157 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:09.712410 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.712384 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:23:09.712569 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.712508 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:23:09.800348 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.800313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:23:09.802847 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.802828 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:23:09.813155 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.813132 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:23:09.823571 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.823546 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:23:09.930838 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.930808 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9pl6q\"" Nov 01 22:23:09.938478 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.938458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:23:09.942992 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.942965 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fpxpg\"" Nov 01 22:23:09.951185 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.951164 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:23:09.960799 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.960712 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5scqq\"" Nov 01 22:23:09.968857 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:09.968837 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:23:10.215015 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:10.214901 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8cbk8"] Nov 01 22:23:10.217543 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:23:10.217516 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21a7a1c_6db6_422d_a24f_5bdc6b4a700c.slice/crio-cbd8015f1ffb7ea0b632a028ed80b82cf572c9c7cc67514a94c4f2095bdd4053 WatchSource:0}: Error finding container cbd8015f1ffb7ea0b632a028ed80b82cf572c9c7cc67514a94c4f2095bdd4053: Status 404 returned error can't find the container with id cbd8015f1ffb7ea0b632a028ed80b82cf572c9c7cc67514a94c4f2095bdd4053 Nov 01 22:23:10.253358 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:10.253330 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cg4rv"] Nov 01 22:23:10.256034 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:23:10.255995 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8739003e_fba1_4d47_80e9_0f484d117b73.slice/crio-1c2c3998f2f3e90dc73190e02bb652ae7a42c289d3d7c92a114eab460e226ccc WatchSource:0}: Error finding container 1c2c3998f2f3e90dc73190e02bb652ae7a42c289d3d7c92a114eab460e226ccc: Status 404 returned error can't find the container with id 1c2c3998f2f3e90dc73190e02bb652ae7a42c289d3d7c92a114eab460e226ccc Nov 01 22:23:10.266331 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:10.266248 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vzk9n"] Nov 01 22:23:10.268653 ip-10-0-134-88 kubenswrapper[2580]: W1101 22:23:10.268626 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbe2700_733d_4ebd_9f47_7b9f30bb62fd.slice/crio-2be4be0e9cfc3c8dedfe37691dab2da955e4cd5e14fc1a5cee7b9104f11586e0 WatchSource:0}: Error finding container 2be4be0e9cfc3c8dedfe37691dab2da955e4cd5e14fc1a5cee7b9104f11586e0: Status 404 returned error can't find the container with id 2be4be0e9cfc3c8dedfe37691dab2da955e4cd5e14fc1a5cee7b9104f11586e0 Nov 01 22:23:10.416164 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:10.416131 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzk9n" event={"ID":"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd","Type":"ContainerStarted","Data":"2be4be0e9cfc3c8dedfe37691dab2da955e4cd5e14fc1a5cee7b9104f11586e0"} Nov 01 22:23:10.417275 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:10.417247 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cg4rv" event={"ID":"8739003e-fba1-4d47-80e9-0f484d117b73","Type":"ContainerStarted","Data":"1c2c3998f2f3e90dc73190e02bb652ae7a42c289d3d7c92a114eab460e226ccc"} Nov 01 22:23:10.418479 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:10.418456 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8cbk8" event={"ID":"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c","Type":"ContainerStarted","Data":"cbd8015f1ffb7ea0b632a028ed80b82cf572c9c7cc67514a94c4f2095bdd4053"} Nov 01 22:23:11.660308 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:11.660269 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:11.660785 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:11.660344 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:12.431722 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:12.431658 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8cbk8" event={"ID":"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c","Type":"ContainerStarted","Data":"fcce9cb0d0af4e73cd65c0749d4d720189f00cba0f2e18236e56f72129c00be8"} Nov 01 22:23:12.431912 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:12.431756 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8cbk8" event={"ID":"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c","Type":"ContainerStarted","Data":"03b423c18d32320444f279929130ab5eeb9e2dc9e9139313587c0437412b05f7"} Nov 01 22:23:12.448595 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:12.448531 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8cbk8" podStartSLOduration=66.771751375 podStartE2EDuration="1m8.44851277s" totalImagesPullingTime="1.676761395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:12.446929176 +0000 UTC m=+68.973856965" watchObservedRunningTime="2026-11-01 22:23:12.44851277 +0000 UTC m=+68.975440537" Nov 01 22:23:14.186022 ip-10-0-134-88 kubenswrapper[2580]: E1101 22:23:14.185988 2580 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:23:14.441911 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:14.441833 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vzk9n" event={"ID":"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd","Type":"ContainerStarted","Data":"630fddad5c1092d415b85da8b6cbd1646fa1090972a00bc6a0afb6d198f2eae0"} Nov 01 22:23:14.442070 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:14.441958 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:23:14.458206 ip-10-0-134-88 kubenswrapper[2580]: I1101 22:23:14.458145 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vzk9n" podStartSLOduration=67.22525945 podStartE2EDuration="1m10.458126927s" totalImagesPullingTime="3.232867477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:14.456298042 +0000 UTC m=+70.983225808" watchObservedRunningTime="2026-11-01 22:23:14.458126927 +0000 UTC m=+70.985054685" Nov 01 22:23:16.150818 ip-10-0-134-88 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 22:23:16.166557 ip-10-0-134-88 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 22:23:16.166732 ip-10-0-134-88 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 22:23:16.166901 ip-10-0-134-88 systemd[1]: kubelet.service: Consumed 4.300s CPU time. Nov 01 22:23:16.183684 ip-10-0-134-88 systemd[1]: Starting Kubernetes Kubelet... Nov 01 22:23:16.245892 ip-10-0-134-88 kubenswrapper[6466]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:16.245892 ip-10-0-134-88 kubenswrapper[6466]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 22:23:16.245892 ip-10-0-134-88 kubenswrapper[6466]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:16.245892 ip-10-0-134-88 kubenswrapper[6466]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:16.248287 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248204 6466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:16.248287 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248287 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:16.248287 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248291 6466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248295 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248297 6466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248300 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248302 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248306 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248314 6466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248317 6466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248320 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248323 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248325 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248328 6466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248331 6466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248333 6466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248336 6466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248338 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248341 6466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248344 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248346 6466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:16.248384 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248349 6466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248351 6466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248354 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248357 6466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248359 6466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248362 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248365 6466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248367 6466 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248370 6466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248372 6466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248375 6466 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248377 6466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248380 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248383 6466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248386 6466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248389 6466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248392 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248395 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248398 6466 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248401 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:16.248881 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248405 6466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248408 6466 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248410 6466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248413 6466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248417 6466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248422 6466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248425 6466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248428 6466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248430 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248433 6466 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248436 6466 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248438 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248440 6466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248443 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248447 6466 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248450 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248453 6466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248455 6466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248458 6466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248461 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:16.249397 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248463 6466 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248466 6466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248468 6466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248471 6466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248473 6466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248481 6466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248484 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248487 6466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248489 6466 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248492 6466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248495 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248497 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248500 6466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248504 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248506 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248509 6466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248511 6466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248514 6466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248516 6466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:16.250180 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248519 6466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248521 6466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248524 6466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248527 6466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248530 6466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248534 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248537 6466 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248541 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248544 6466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248546 6466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248549 6466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248552 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248555 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248557 6466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248559 6466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248562 6466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248565 6466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248567 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248570 6466 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:16.250762 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248572 6466 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248575 6466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248577 6466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248582 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248584 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248586 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.248589 6466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249840 6466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249857 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249862 6466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249866 6466 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249871 6466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249876 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249881 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249892 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249896 6466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249900 6466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249905 6466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249910 6466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:16.251274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249914 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249919 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249923 6466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249928 6466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249932 6466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249937 6466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249945 6466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249955 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249960 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249967 6466 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249971 6466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249975 6466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249980 6466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249985 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249990 6466 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249994 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.249998 6466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250003 6466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250007 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:16.251790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250016 6466 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250020 6466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250025 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250030 6466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250035 6466 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250039 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250044 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250048 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250053 6466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250057 6466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250537 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250660 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250665 6466 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250668 6466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250671 6466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250680 6466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250683 6466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250685 6466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250688 6466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:16.252381 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250691 6466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250694 6466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250697 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250699 6466 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250702 6466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250705 6466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250707 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250710 6466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250713 6466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250715 6466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250718 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250721 6466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250724 6466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250727 6466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250729 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250732 6466 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250734 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250737 6466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250740 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250742 6466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:16.252878 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250744 6466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250747 6466 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250751 6466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250754 6466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250757 6466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250759 6466 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250763 6466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250766 6466 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250770 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250773 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250776 6466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250779 6466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250781 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250784 6466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250786 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250789 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250792 6466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250794 6466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250797 6466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250800 6466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:16.253444 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250802 6466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250805 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250807 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250810 6466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250813 6466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250815 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250818 6466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250821 6466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250823 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250826 6466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250828 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250831 6466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250838 6466 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250841 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250844 6466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.250850 6466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251411 6466 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251418 6466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251421 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251425 6466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:16.253950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251429 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251431 6466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251434 6466 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251437 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251440 6466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251443 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251446 6466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251448 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251451 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251454 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251456 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251459 6466 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251463 6466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251467 6466 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251471 6466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251476 6466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251479 6466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251482 6466 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251485 6466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:16.254471 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251487 6466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251491 6466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251493 6466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251496 6466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251498 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251503 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251506 6466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251508 6466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251511 6466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251513 6466 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251516 6466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251519 6466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251521 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251524 6466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251526 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251529 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251532 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251535 6466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251537 6466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:16.254950 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251540 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251543 6466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251545 6466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251548 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251550 6466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251553 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251555 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251558 6466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251560 6466 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251563 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251567 6466 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251569 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251572 6466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251574 6466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251577 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251581 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251583 6466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251586 6466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251590 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:16.255636 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251593 6466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251595 6466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251598 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251600 6466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251603 6466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251605 6466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251608 6466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251610 6466 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251613 6466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251615 6466 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251618 6466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251620 6466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251623 6466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251625 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251629 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251632 6466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251634 6466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251637 6466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251639 6466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251642 6466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:16.256493 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251644 6466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251647 6466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251649 6466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251652 6466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251655 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251658 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251660 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251663 6466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251665 6466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251668 6466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251670 6466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251674 6466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251678 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251680 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251683 6466 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251686 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251688 6466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251691 6466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251693 6466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:16.257464 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251696 6466 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251698 6466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251700 6466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251703 6466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251705 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.251708 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251798 6466 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251806 6466 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251817 6466 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251822 6466 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251826 6466 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251829 6466 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251833 6466 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251838 6466 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251841 6466 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251844 6466 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251848 6466 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251851 6466 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251855 6466 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251858 6466 flags.go:64] FLAG: --cgroup-root="" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251860 6466 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251864 6466 flags.go:64] FLAG: --client-ca-file="" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251867 6466 flags.go:64] FLAG: --cloud-provider="external" Nov 01 22:23:16.258311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251870 6466 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251878 6466 flags.go:64] FLAG: --cluster-domain="" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251881 6466 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251884 6466 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251888 6466 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251892 6466 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251896 6466 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251899 6466 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251902 6466 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251905 6466 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251909 6466 flags.go:64] FLAG: --contention-profiling="false" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251912 6466 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251915 6466 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251918 6466 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251920 6466 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251925 6466 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251932 6466 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251935 6466 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251939 6466 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251942 6466 flags.go:64] FLAG: --enable-server="true" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251945 6466 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251949 6466 flags.go:64] FLAG: --event-burst="100" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251952 6466 flags.go:64] FLAG: --event-qps="50" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251955 6466 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 22:23:16.259354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251958 6466 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251961 6466 flags.go:64] FLAG: --eviction-hard="" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251966 6466 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251969 6466 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251972 6466 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251975 6466 flags.go:64] FLAG: --eviction-soft="" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251978 6466 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251981 6466 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251984 6466 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251988 6466 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251991 6466 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251994 6466 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.251997 6466 flags.go:64] FLAG: --feature-gates="" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252002 6466 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252005 6466 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252009 6466 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252012 6466 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252015 6466 flags.go:64] FLAG: --healthz-port="10248" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252018 6466 flags.go:64] FLAG: --help="false" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252021 6466 flags.go:64] FLAG: --hostname-override="ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252024 6466 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252027 6466 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252030 6466 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 22:23:16.260502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252034 6466 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252037 6466 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252040 6466 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252043 6466 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252047 6466 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252050 6466 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252053 6466 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252056 6466 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252059 6466 flags.go:64] FLAG: --kube-reserved="" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252062 6466 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252067 6466 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252070 6466 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252073 6466 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252076 6466 flags.go:64] FLAG: --lock-file="" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252078 6466 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252081 6466 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252085 6466 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252090 6466 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252094 6466 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252097 6466 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252100 6466 flags.go:64] FLAG: --logging-format="text" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252118 6466 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252123 6466 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252126 6466 flags.go:64] FLAG: --manifest-url="" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252129 6466 flags.go:64] FLAG: --manifest-url-header="" Nov 01 22:23:16.261522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252133 6466 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252136 6466 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252140 6466 flags.go:64] FLAG: --max-pods="110" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252143 6466 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252146 6466 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252149 6466 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252153 6466 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252156 6466 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252159 6466 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252162 6466 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252169 6466 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252172 6466 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252176 6466 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252179 6466 flags.go:64] FLAG: --pod-cidr="" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252182 6466 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252185 6466 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252188 6466 flags.go:64] FLAG: --pods-per-core="0" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252197 6466 flags.go:64] FLAG: --port="10250" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252200 6466 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252203 6466 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d733dfd78c2c1aca" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252206 6466 flags.go:64] FLAG: --qos-reserved="" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252209 6466 flags.go:64] FLAG: --read-only-port="10255" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252212 6466 flags.go:64] FLAG: --register-node="true" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252216 6466 flags.go:64] FLAG: --register-with-taints="" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252220 6466 flags.go:64] FLAG: --registry-burst="10" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252224 6466 flags.go:64] FLAG: --registry-qps="5" Nov 01 22:23:16.262613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252227 6466 flags.go:64] FLAG: --reserved-cpus="" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252230 6466 flags.go:64] FLAG: --reserved-memory="" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252234 6466 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252237 6466 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252240 6466 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252243 6466 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252246 6466 flags.go:64] FLAG: --runonce="false" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252249 6466 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252252 6466 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252255 6466 flags.go:64] FLAG: --seccomp-default="false" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252258 6466 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252261 6466 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252264 6466 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252267 6466 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252270 6466 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252273 6466 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252276 6466 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252279 6466 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252282 6466 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252286 6466 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252289 6466 flags.go:64] FLAG: --system-cgroups="" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252291 6466 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252297 6466 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252301 6466 flags.go:64] FLAG: --tls-cert-file="" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252304 6466 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 22:23:16.263837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252310 6466 flags.go:64] FLAG: --tls-min-version="" Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252313 6466 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252316 6466 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252319 6466 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252322 6466 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252325 6466 flags.go:64] FLAG: --v="2" Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252330 6466 flags.go:64] FLAG: --version="false" Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252335 6466 flags.go:64] FLAG: --vmodule="" Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252339 6466 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252342 6466 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252478 6466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252482 6466 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252485 6466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252489 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252492 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252494 6466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252497 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252499 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252502 6466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252504 6466 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252507 6466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252510 6466 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:16.264861 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252512 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252515 6466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252518 6466 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252520 6466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252523 6466 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252526 6466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252528 6466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252531 6466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252535 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252538 6466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252540 6466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252543 6466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252546 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252548 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252551 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252554 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252557 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252560 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252562 6466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252565 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:16.265631 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252567 6466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252570 6466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252573 6466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252575 6466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252578 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252580 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252583 6466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252585 6466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252588 6466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252590 6466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252593 6466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252595 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252599 6466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252602 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252605 6466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252608 6466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252612 6466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252615 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:16.266204 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252618 6466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252621 6466 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252625 6466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252628 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252630 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252633 6466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252635 6466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252638 6466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252640 6466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252643 6466 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252646 6466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252648 6466 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252651 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252653 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252656 6466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252658 6466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252661 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252663 6466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252666 6466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252668 6466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:16.266755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252671 6466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252673 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252676 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252678 6466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252681 6466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252684 6466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252686 6466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252689 6466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252691 6466 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252694 6466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252697 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252699 6466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252702 6466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252712 6466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252717 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252720 6466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252723 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252726 6466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252728 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252731 6466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:16.267274 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252733 6466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252736 6466 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252739 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252741 6466 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252744 6466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252746 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252749 6466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252752 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252754 6466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252757 6466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252759 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252762 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252764 6466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252767 6466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252769 6466 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252772 6466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:16.267768 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.252799 6466 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252834 6466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252839 6466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252842 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252846 6466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252849 6466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252851 6466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252854 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252857 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252860 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252865 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252868 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252871 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252874 6466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252876 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252880 6466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252883 6466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252886 6466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252889 6466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252892 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:16.268184 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252894 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252897 6466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252899 6466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252902 6466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252905 6466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252907 6466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252910 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252914 6466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252918 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252921 6466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252924 6466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252928 6466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252932 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252936 6466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252939 6466 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252941 6466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252944 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252947 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:16.268790 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252949 6466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252952 6466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252954 6466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252957 6466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252961 6466 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252965 6466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252968 6466 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252970 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252973 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252977 6466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252979 6466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252982 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252985 6466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252987 6466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252990 6466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252993 6466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252995 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.252998 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253001 6466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253004 6466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:16.269380 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253006 6466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253009 6466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253011 6466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253014 6466 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253017 6466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253019 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253022 6466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253024 6466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253026 6466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253029 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253032 6466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253034 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253037 6466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253039 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253042 6466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253044 6466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253049 6466 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253052 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253054 6466 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253057 6466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:16.269871 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253060 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253064 6466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253066 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253069 6466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253071 6466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253074 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253076 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253079 6466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253082 6466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253084 6466 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253087 6466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253090 6466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253093 6466 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253095 6466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253098 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253100 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253119 6466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253123 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253126 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253128 6466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:16.270383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253131 6466 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253134 6466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253136 6466 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253139 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253143 6466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253145 6466 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253148 6466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253151 6466 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.253155 6466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.257834 6466 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.257856 6466 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.257882 6466 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.257890 6466 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.257974 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.257981 6466 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.257987 6466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.257992 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.257997 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258001 6466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258005 6466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:16.270857 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258010 6466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258014 6466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258019 6466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258023 6466 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258029 6466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258037 6466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258043 6466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258047 6466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258052 6466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258057 6466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258061 6466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258066 6466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258070 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258075 6466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258079 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258083 6466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258088 6466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258093 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258097 6466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:16.271391 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258101 6466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258132 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258138 6466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258144 6466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258149 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258157 6466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258162 6466 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258166 6466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258171 6466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258175 6466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258179 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258184 6466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258188 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258193 6466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258197 6466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258202 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258207 6466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258211 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258216 6466 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:16.272160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258220 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258224 6466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258228 6466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258233 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258237 6466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258242 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258246 6466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258250 6466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258254 6466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258258 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258262 6466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258266 6466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258270 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258275 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258279 6466 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258285 6466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258289 6466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258295 6466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258299 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:16.272908 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258303 6466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258308 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258312 6466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258316 6466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258320 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258325 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258329 6466 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258333 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258337 6466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258341 6466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258345 6466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258350 6466 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258355 6466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258359 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258363 6466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258368 6466 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258372 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258377 6466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258381 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258385 6466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:16.273516 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258389 6466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258394 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258399 6466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258404 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258408 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258413 6466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258417 6466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258421 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258426 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258431 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258435 6466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258441 6466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258445 6466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258450 6466 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258454 6466 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258459 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258463 6466 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258467 6466 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258471 6466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258476 6466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:16.274241 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258480 6466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258485 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.258530 6466 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258761 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258769 6466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258774 6466 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258779 6466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258783 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258788 6466 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258792 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258797 6466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258801 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258805 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258810 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258814 6466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258819 6466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:16.274876 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258823 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258827 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258832 6466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258836 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258841 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258846 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258850 6466 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258854 6466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258860 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258864 6466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258869 6466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258873 6466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258877 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258881 6466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258886 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258890 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258894 6466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258898 6466 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258903 6466 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:16.275484 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258907 6466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258911 6466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258916 6466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258920 6466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258924 6466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258928 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258933 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258937 6466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258941 6466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258946 6466 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258950 6466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258955 6466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258959 6466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258963 6466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258968 6466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258973 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258979 6466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258985 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258990 6466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258995 6466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:16.276140 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.258999 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259003 6466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259008 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259013 6466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259017 6466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259021 6466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259026 6466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259030 6466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259034 6466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259038 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259042 6466 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259047 6466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259051 6466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259056 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259060 6466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259064 6466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259069 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259073 6466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259077 6466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259082 6466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:16.276646 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259087 6466 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259091 6466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259095 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259099 6466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259119 6466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259125 6466 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259129 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259133 6466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259137 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259145 6466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259148 6466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259153 6466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259157 6466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259161 6466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259166 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259171 6466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259176 6466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259180 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259185 6466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259188 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:16.277182 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259193 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259197 6466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259201 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259205 6466 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259212 6466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259219 6466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259224 6466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259229 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259233 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259238 6466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259245 6466 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259249 6466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259254 6466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259258 6466 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.259299 6466 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:16.277681 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259354 6466 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259360 6466 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259365 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259369 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259374 6466 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259378 6466 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259383 6466 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259389 6466 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259394 6466 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259398 6466 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259402 6466 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259407 6466 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259412 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259420 6466 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259426 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259431 6466 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259435 6466 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259440 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259444 6466 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259448 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:16.278058 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259453 6466 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259457 6466 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259461 6466 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259465 6466 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259470 6466 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259474 6466 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259478 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259483 6466 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259487 6466 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259492 6466 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259496 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259501 6466 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259505 6466 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259509 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259514 6466 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259518 6466 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259522 6466 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259526 6466 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259531 6466 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259537 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:16.278568 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259541 6466 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259546 6466 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259550 6466 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259554 6466 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259559 6466 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259563 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259568 6466 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259573 6466 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259577 6466 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259581 6466 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259586 6466 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259590 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259594 6466 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259598 6466 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259603 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259607 6466 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259611 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259615 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259620 6466 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259624 6466 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:16.279148 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259629 6466 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259636 6466 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259642 6466 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259648 6466 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259654 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259659 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259664 6466 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259668 6466 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259673 6466 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259678 6466 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259682 6466 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259688 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259692 6466 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259696 6466 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259701 6466 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259705 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259710 6466 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259714 6466 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259718 6466 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:16.279637 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259724 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259728 6466 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259732 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259736 6466 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259741 6466 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259745 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259750 6466 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259754 6466 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259758 6466 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259764 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259768 6466 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259772 6466 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259777 6466 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259781 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259786 6466 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259790 6466 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259794 6466 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259799 6466 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259803 6466 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:16.280128 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259808 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259812 6466 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259816 6466 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259821 6466 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259825 6466 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259833 6466 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259837 6466 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:16.259841 6466 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.259929 6466 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.261551 6466 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.261636 6466 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.261905 6466 server.go:1014] "Starting client certificate rotation" Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.262032 6466 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.262088 6466 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:03 +0000 UTC" deadline="2028-05-22 00:20:43.930826617 +0000 UTC" Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.262123 6466 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13609h57m27.668705348s" Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.262661 6466 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:23:16.280639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.264368 6466 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:23:16.281134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.266051 6466 log.go:25] "Validated CRI v1 runtime API" Nov 01 22:23:16.281134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.268043 6466 log.go:25] "Validated CRI v1 image API" Nov 01 22:23:16.281134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.268836 6466 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 22:23:16.281134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.271049 6466 fs.go:135] Filesystem UUIDs: map[0b2edfcf-f608-4439-a77e-9461fdc3e9f4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7b6f287e-d3a8-4b7d-ac35-0cf118394913:/dev/nvme0n1p3] Nov 01 22:23:16.281380 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.271069 6466 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b8931f969e4f432f5b9512dc174708eefdf8a4e50e264ee70b00cb41e3bd8a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b8931f969e4f432f5b9512dc174708eefdf8a4e50e264ee70b00cb41e3bd8a6/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1629c22a269b002d6230d7524a8ff25fa938d89393541ddd3513edd6ea73953a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1629c22a269b002d6230d7524a8ff25fa938d89393541ddd3513edd6ea73953a/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a8667518758b9b4b88869f84365d55fc37b8e0deecc74895a4a13469dda1263/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a8667518758b9b4b88869f84365d55fc37b8e0deecc74895a4a13469dda1263/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1aea23dcf3e0b02275ede97637ab172be1c6bfd2b9618d3aac9b2bc5f11acaa4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1aea23dcf3e0b02275ede97637ab172be1c6bfd2b9618d3aac9b2bc5f11acaa4/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c2c3998f2f3e90dc73190e02bb652ae7a42c289d3d7c92a114eab460e226ccc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c2c3998f2f3e90dc73190e02bb652ae7a42c289d3d7c92a114eab460e226ccc/userdata/shm major:0 minor:444 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/288cef5dfe34637db0488d39d62c0703785a30b8abd5ab890a2a54d4254c5097/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/288cef5dfe34637db0488d39d62c0703785a30b8abd5ab890a2a54d4254c5097/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2aff8080095aa44f043614b07c3f72b0d828a4c9db426dd5b46847e31f12daf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2aff8080095aa44f043614b07c3f72b0d828a4c9db426dd5b46847e31f12daf2/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2be4be0e9cfc3c8dedfe37691dab2da955e4cd5e14fc1a5cee7b9104f11586e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2be4be0e9cfc3c8dedfe37691dab2da955e4cd5e14fc1a5cee7b9104f11586e0/userdata/shm major:0 minor:446 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46844a8e32c7b4cfa6f04a52ced4f17086e0311f0f1d2fa694589b7525f975b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46844a8e32c7b4cfa6f04a52ced4f17086e0311f0f1d2fa694589b7525f975b8/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b75527010af335f6bc91af969b0d3f5a9ec9ae1caf1ec6b95d5c1224420b98f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b75527010af335f6bc91af969b0d3f5a9ec9ae1caf1ec6b95d5c1224420b98f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74dba1cf8098e916cca03e7c5c84cae0255d622afb2a200ada024e5531890363/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74dba1cf8098e916cca03e7c5c84cae0255d622afb2a200ada024e5531890363/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c6fe3b2750890436a0c7e02358d55145f299e80ba2ca1f043b8d75ac056398b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c6fe3b2750890436a0c7e02358d55145f299e80ba2ca1f043b8d75ac056398b/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98cd666dca399b17251fe1448ac855173ee885cc270031f12268dfb12b10bad9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98cd666dca399b17251fe1448ac855173ee885cc270031f12268dfb12b10bad9/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ff711848f297a3eba2d1e200eb8c14f77a2569882c909d8515bb893f7830428/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ff711848f297a3eba2d1e200eb8c14f77a2569882c909d8515bb893f7830428/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a861487740395d771feca30986d9bd4c21af987fe32b4235c30635cd0547c5b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a861487740395d771feca30986d9bd4c21af987fe32b4235c30635cd0547c5b9/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c31bacab52a6216b2f7a96ce3d9b44efe0246fcfd576ca4c35a77be8565ac4b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c31bacab52a6216b2f7a96ce3d9b44efe0246fcfd576ca4c35a77be8565ac4b6/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c478781b15d1eddbed88055f52b54ff23b89ea0b3c980619a6d32e795b310fc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c478781b15d1eddbed88055f52b54ff23b89ea0b3c980619a6d32e795b310fc6/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbd8015f1ffb7ea0b632a028ed80b82cf572c9c7cc67514a94c4f2095bdd4053/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbd8015f1ffb7ea0b632a028ed80b82cf572c9c7cc67514a94c4f2095bdd4053/userdata/shm major:0 minor:442 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dee8af8b6e9db4d10d99cdaf32ee9aa115336aa79d884d90bab5fe69811619ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dee8af8b6e9db4d10d99cdaf32ee9aa115336aa79d884d90bab5fe69811619ce/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df30684be22fdfc276dbba87c25d24c6e68907c5da7ff9184d651db76991a3f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df30684be22fdfc276dbba87c25d24c6e68907c5da7ff9184d651db76991a3f1/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e04aadde8ef47903434fc2edc9676958a2fd296216f41ff779108dc3ab8d7286/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e04aadde8ef47903434fc2edc9676958a2fd296216f41ff779108dc3ab8d7286/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eda1485c80c9159bf77bb63e96a59ce816c02b7f3b84c283e36cdcb5595df434/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eda1485c80c9159bf77bb63e96a59ce816c02b7f3b84c283e36cdcb5595df434/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2c80714eec266084edc75df0ba27e03ab2ab3216ff8d03ee43f1343555934e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2c80714eec266084edc75df0ba27e03ab2ab3216ff8d03ee43f1343555934e5/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~projected/kube-api-access-lm9zh:{mountpoint:/var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~projected/kube-api-access-lm9zh major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26366d8c-76f1-479e-9715-6512f20bd3c6/volumes/kubernetes.io~projected/kube-api-access-ctgxg:{mountpoint:/var/lib/kubelet/pods/26366d8c-76f1-479e-9715-6512f20bd3c6/volumes/kubernetes.io~projected/kube-api-access-ctgxg major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26366d8c-76f1-479e-9715-6512f20bd3c6/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/26366d8c-76f1-479e-9715-6512f20bd3c6/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d408013-e118-42d1-809f-5d3b50f12565/volumes/kubernetes.io~projected/kube-api-access-j7sqc:{mountpoint:/var/lib/kubelet/pods/4d408013-e118-42d1-809f-5d3b50f12565/volumes/kubernetes.io~projected/kube-api-access-j7sqc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd/volumes/kubernetes.io~projected/kube-api-access-7w5t7:{mountpoint:/var/lib/kubelet/pods/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd/volumes/kubernetes.io~projected/kube-api-access-7w5t7 major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5baeb178-3616-4b1d-8515-d8ff71486666/volumes/kubernetes.io~projected/kube-api-access-qrbgq:{mountpoint:/var/lib/kubelet/pods/5baeb178-3616-4b1d-8515-d8ff71486666/volumes/kubernetes.io~projected/kube-api-access-qrbgq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6962a1b2-43de-4da8-a1f4-af8101d42fd0/volumes/kubernetes.io~projected/kube-api-access-lsfjt:{mountpoint:/var/lib/kubelet/pods/6962a1b2-43de-4da8-a1f4-af8101d42fd0/volumes/kubernetes.io~projected/kube-api-access-lsfjt major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d/volumes/kubernetes.io~projected/kube-api-access-6fhtq:{mountpoint:/var/lib/kubelet/pods/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d/volumes/kubernetes.io~projected/kube-api-access-6fhtq major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76a5a648-0585-408d-8556-c0e31dcea942/volumes/kubernetes.io~projected/kube-api-access-2hsxj:{mountpoint:/var/lib/kubelet/pods/76a5a648-0585-408d-8556-c0e31dcea942/volumes/kubernetes.io~projected/kube-api-access-2hsxj major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~projected/kube-api-access-44ld4:{mountpoint:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~projected/kube-api-access-44ld4 major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8739003e-fba1-4d47-80e9-0f484d117b73/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8739003e-fba1-4d47-80e9-0f484d117b73/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c/volumes/kubernetes.io~projected/kube-api-access-9wfck:{mountpoint:/var/lib/kubelet/pods/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c/volumes/kubernetes.io~projected/kube-api-access-9wfck major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c/volumes/kubernetes.io~projected/kube-api-access-qjpf6:{mountpoint:/var/lib/kubelet/pods/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c/volumes/kubernetes.io~projected/kube-api-access-qjpf6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~projected/kube-api-access-4sgm9:{mountpoint:/var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~projected/kube-api-access-4sgm9 major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0/volumes/kubernetes.io~projected/kube-api-access-h5csc:{mountpoint:/var/lib/kubelet/pods/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0/volumes/kubernetes.io~projected/kube-api-access-h5csc major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d77255dd-93d5-4c30-8a4c-f4c7ea44d702/volumes/kubernetes.io~projected/kube-api-access-bntx6:{mountpoint:/var/lib/kubelet/pods/d77255dd-93d5-4c30-8a4c-f4c7ea44d702/volumes/kubernetes.io~projected/kube-api-access-bntx6 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d77255dd-93d5-4c30-8a4c-f4c7ea44d702/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d77255dd-93d5-4c30-8a4c-f4c7ea44d702/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d77255dd-93d5-4c30-8a4c-f4c7ea44d702/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d77255dd-93d5-4c30-8a4c-f4c7ea44d702/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1baadc8-f14c-490f-9655-a55352e7eb8b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e1baadc8-f14c-490f-9655-a55352e7eb8b/volumes/kubernetes.io~secret/agent-certs major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1dd1e50-ecd0-4724-862d-752e9e1aecd0/volumes/kubernetes.io~projected/kube-api-access-k5dtj:{mountpoint:/var/lib/kubelet/pods/e1dd1e50-ecd0-4724-862d-752e9e1aecd0/volumes/kubernetes.io~projected/kube-api-access-k5dtj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8572e79-8cf8-4b15-b77f-ba409bd3b51f/volumes/kubernetes.io~projected/kube-api-access-c577v:{mountpoint:/var/lib/kubelet/pods/e8572e79-8cf8-4b15-b77f-ba409bd3b51f/volumes/kubernetes.io~projected/kube-api-access-c577v major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8572e79-8cf8-4b15-b77f-ba409bd3b51f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e8572e79-8cf8-4b15-b77f-ba409bd3b51f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8572e79-8cf8-4b15-b77f-ba409bd3b51f/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/e8572e79-8cf8-4b15-b77f-ba409bd3b51f/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9ed2868-24ad-473c-bac4-01344c0e4bed/volumes/kubernetes.io~projected/kube-api-access-c2r9f:{mountpoint:/var/lib/kubelet/pods/e9ed2868-24ad-473c-bac4-01344c0e4bed/volumes/kubernetes.io~projected/kube-api-access-c2r9f major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9ed2868-24ad-473c-bac4-01344c0e4bed/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e9ed2868-24ad-473c-bac4-01344c0e4bed/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9ed2868-24ad-473c-bac4-01344c0e4bed/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/e9ed2868-24ad-473c-bac4-01344c0e4bed/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea837196-594f-42a6-981b-74717f55e790/volumes/kubernetes.io~projected/kube-api-access-qkns2:{mountpoint:/var/lib/kubelet/pods/ea837196-594f-42a6-981b-74717f55e790/volumes/kubernetes.io~projected/kube-api-access-qkns2 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/web-config/alertmanager/9 major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/web-config/config-reloader/6 major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~empty-dir/config-out major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~projected/kube-api-access-78lnx:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~projected/kube-api-access-78lnx major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~projected/tls-assets major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/config-volume major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/web-config major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2c4e933-b200-47a6-94c8-1966aeb16884/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f2c4e933-b200-47a6-94c8-1966aeb16884/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2c4e933-b200-47a6-94c8-1966aeb16884/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f2c4e933-b200-47a6-94c8-1966aeb16884/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2c4e933-b200-47a6-94c8-1966aeb16884/volumes/kubernetes.io~projected/kube-api-access-nt6gp:{mountpoint:/var/lib/kubelet/pods/f2c4e933-b200-47a6-94c8-1966aeb16884/volumes/kubernetes.io~projected/kube-api-access-nt6gp major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/20971c964c4303fb83e3dc7fbf8e2a055da01451a5ae3063ff7a8897e4f922b9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a0ddabee8de603cbcc06c2d912ffb2852a7dc31a542f00c91090be7896b3bf79/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/865a71475923c1edccedcf181a3c69154d5a8e657009572421fa9f7e8068a038/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/2d2ff2ebe766c97b16b5f1a822087dc3768f223846ccc0b006eefed2f945bf8b/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/31d75e64dfb73f597619a397b0fc9e0801145bc0ecf740cee154694d1c1e1ffa/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4f0138204afac73a0cc306628969e4f582bb8262ca76a2ccef6052afd8d33943/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/50eaa445f987d2cfb989b20ec16b04df9d9b283fbf0271cf1679c81921cff55d/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d6801784e8c97e752400f2621bff93f70e029f3da8807cd7dc8746f22f447fe9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/08c5e49e278956c009a5292e0dffb05d1dd6511e0ecd5c1e1109a82a16c661c5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/310ef7e931b0ea9f5bfd32b30a72a9a232671767beaddaeea9be6a71e73c9509/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/19c0260fa8c351d23f1800efc53520921cd61a255429523d74746c8253ab83d2/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/871c8d8086e6e1240c80b4096a4ffffea928fa6ac4bb4df2eaeebb7c49eeeb0c/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/55e8f0e1cfef54a399d3f57c8454dc8a2e38dcdc4a1b452734c3b25ffd05ec01/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/6a0274f555444b19ecfdfa5c7a4fa6c57bff2154053776e3a000ec2746b609e7/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bdf255bb8b0cb47009afa32d3ccc17d4e7bb20774a6bb70bf91972a360b5f502/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/439fcad5d2db15cc549b250280f44ece5a5280dcafa8bc8c55505766e14fce03/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6b54d795cd1818093396fe4b487091ade3f1713ffb4495c7c564158187327701/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8a988e706f717acd2b61f4de5df62a155164767590c01e78a865ae12ab0db214/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/928f6c4329c9ce419c60d1391ebb3b3def18c212ec0c118221c1218906557251/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/577bd0e3aaa8dbac757e5a777a12afa8c4af005d299bb76463117748ea8b5c2c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8971624526c7719c6a6163bb9e259289479111f44d2acfab60c3bf8f3a8b2f07/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/416e44cefafa2ec1864544ab99ff34cae42c11e40ca8170a746d280fb1e5a21f/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/855c42ae8f1242d258ff0418c541011e362ad6a39da9dcee76891938a056410c/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/85c99d4eeee59ec4cc3a0c53f289e32c0b579520c4cce59d95a3bb914f8349d8/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/6df7760e370d0621aeae5d77d83fc3872bb2897588f6ad79a260740746764d3f/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/f09b22c48555c66b3c19109cff71e4ab7b866cbd24c2d0c75deaee3ad3b12448/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/66253992ad65db9e5ae1be310b3084f81272a2f3eeb470e513171bc82b7c3a0a/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/c65786938a80c2b6a7e28d98ce6e942f2c6cc7e4d1bd7f62750513ca55fc55fe/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/e091445be34ab47851b3b3e90f4fdcd70cb668b05b75b60fe854e9e2fda27149/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/aafedb53ff6a5f7ec389419314eb760db270ace9480b65e45b7cc99ee56a1e63/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/b4017c41394e57159a229b3f79e8a07ad1c861a0c92febf11caa337ec7ca2fa1/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/de5af44bf24f8424e86b5eedfa93eb10d1c54f42e71cb27b920ee3fce0cf5d5a/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/1dc574812de079fd92f4491c68b240e8e864c26e8209d206c7ee627260da00ed/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/3d88d3481b05b47858fa9bdc305725ee6aa6d0ef9fb4866ab3a174cb77970ead/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/019f4e4dbd08a726bf685d271da4ffbd4b4279d611f7c25a8e4b110457fa97e4/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/c59952524cff5017254b78fc314a88e062907104f3fb17aec0194f49a7d33492/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/97ce1838a84899b201893ba4c6774436dc45e10b7d1f4168ad8dbb1bfbb30f63/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/bf4c1dc6d24604c1a1f4cb32a53d66f56f23edc1756740699df3315d5639e5d5/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/67e620223faea6f34c2c67d41db390c8345e2077cac46620833e5a9deacc934d/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/007a0d93f8b5a09612416864613b05a3a2b0d3787c81ef4055c7c21cab5e7b06/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/0a295925ce1d4524b13457ac342e0f9e190dd92f421a90e802c250a6aeb6f5a4/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/6ef4daf40e60c5ef0873068358e4a6bae9ed6de671d1c00d3b82429fbe9e506c/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/e44a7a9c1ffe08b2521973c614e4f8ec265fbf69ab66cc80ca449209155e7075/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/9a85dae9737e3185ffeac061180f8e68b6d385cd086ee759b49371212f224e36/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/e90ab88d004ae8698e881839914fdde17043fe765128009df824e052b4c151eb/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/246613d2358c48364f197012b8be2e36bacf379186c70a01500e8d6fa9123862/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/88f072048ba8e95d43763a722c7ed839e7cb872b179aaf76ba0533e14f7b743b/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/91d715b1befba07b25ce89597d5eff56a612f09cda3dd1b4572f47616cf05f1b/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/8d89ae2e1485116297bff3ff34e6f70729fe32117162ca85c052ad78cf83fe68/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/1dd79c1eda3d27c046e89fbb80754b8b33940131ce17f8ac42b686565bb1e542/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/6dc8d1193fe7b5f3c61dbe90d492dae45b743e80565b4e2ecad628218e52d9e6/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/0b0f75d361b185162f174fe1724d0316fc8cfb64c5c93da9be08e5f48182d962/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/95c687aecbfb6910eec3f691a565bdcb5a34f89519e7ee1fb909710d9ca43bf9/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/998f9368695f8edd0aad4cf708b030124538fe5788d1df323becec5c9fd72159/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/06e257eaf30d3f24c27e3f9ca5ae0e9707ec4e1af69d49e44c297e5c7ade5c80/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/599c3bfdd4826b0feead2249575bf7280ffee888e22eb567b37b260a7f73a9e7/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/09c729350787534174178b781e6c5562db6a0e39705c46b33ad0ae79b9556694/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/af4551a40045134d55150ab6cb5c4afebbe418c574f77f983b13af41364a5d5f/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/9171deef353be61d9e3250859198deebe49881a18d1345bb546bf3cf84ddb04c/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/3cd8bf838d72de44a6b6c3fa73ef1b44c57a9ab6330a0e1801f5e43d0f47a282/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/48296e6b7167356a59849e692cc651d13c5e505dae629c521797f4fa13c086b0/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/51e388723137d05b468125bcf51f1f63bc30e16ad14e565f69b8363ed62ea61f/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/e3c89a3f923f44fc5dbadc09f6a72eb9c0209120277a5e78722e1b0e5673504e/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/62d6f64546d93a4bad680c891906a7ba8019f3bb908511fd6ef58b5f84294d06/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/e4299edb5452c303cb94e55f8701cd938647c35eace9897691cb04aeb4b13e1e/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/3b7c587057c65a57ee6c3115a9b9cd823484cc38723430aaa07c00c1a7cc8b78/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/e55a861d3a39c3cd261539827dab2954563b586d38aa5f3598604ba0d6c9ae87/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/9e263e79db530e13919d2bd7b88d09a9f2f3739565c354ee6221f4042e43f1d6/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/624008bdd71aedd44b717fa7b10214605df6c85d19a6d089d7f09cf8f2c2a165/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5963875068ba0fbd7d8d8210502b512c05470b71ea64d13f22cd970191dbe1fb/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/647cc7e4ba8e84e26783b03a4e88a3f0fa837a61a91c1ee0b97e35b9d22bc05d/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/5e97d34d4b32e13eda010bebd2ff1cc8639ff50477c3649f77596f4ee69256f0/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/529072b7c587dcf42d6f4b555b9fbb62cc1e10b16f9406d9cf0266bc35f8acb7/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/99ff0a66e3d1a2d3e55c9330e6c520c4ee2b170cec89af4438b04e25c297b660/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0c099a4e2e83837a7abb9b1b8feadfd0811fbf97490313116dc52a8e0cfcc7a2/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/dc21ed11085cf231f218ed2976a34bcca0f6f4f106e4676df7cd4758712da482/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/bfd180a0d01b48efe7a9824edc2366e5843c7734535b499ed556062bde6117f9/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e06167673a1ae979779503acfca481bf80bec4965efe8d326eb35e91d2dfc64a/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 22:23:16.281819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.280385 6466 manager.go:217] Machine: {Timestamp:2026-11-01 22:23:16.279902321 +0000 UTC m=+0.064341234 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099158 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2250801134e2a781c94211fbac68ea SystemUUID:ec225080-1134-e2a7-81c9-4211fbac68ea BootID:5255c7cb-0ed5-4d44-b1cd-bc10ea7d219b Filesystems:[{Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a861487740395d771feca30986d9bd4c21af987fe32b4235c30635cd0547c5b9/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~projected/kube-api-access-78lnx DeviceMajor:0 DeviceMinor:285 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d77255dd-93d5-4c30-8a4c-f4c7ea44d702/volumes/kubernetes.io~projected/kube-api-access-bntx6 DeviceMajor:0 DeviceMinor:234 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:300 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e04aadde8ef47903434fc2edc9676958a2fd296216f41ff779108dc3ab8d7286/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:440 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/d77255dd-93d5-4c30-8a4c-f4c7ea44d702/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:64 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/288cef5dfe34637db0488d39d62c0703785a30b8abd5ab890a2a54d4254c5097/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b8931f969e4f432f5b9512dc174708eefdf8a4e50e264ee70b00cb41e3bd8a6/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:339 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd/volumes/kubernetes.io~projected/kube-api-access-7w5t7 DeviceMajor:0 DeviceMinor:441 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26366d8c-76f1-479e-9715-6512f20bd3c6/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df30684be22fdfc276dbba87c25d24c6e68907c5da7ff9184d651db76991a3f1/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2aff8080095aa44f043614b07c3f72b0d828a4c9db426dd5b46847e31f12daf2/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74dba1cf8098e916cca03e7c5c84cae0255d622afb2a200ada024e5531890363/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:302 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2c80714eec266084edc75df0ba27e03ab2ab3216ff8d03ee43f1343555934e5/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:278 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c31bacab52a6216b2f7a96ce3d9b44efe0246fcfd576ca4c35a77be8565ac4b6/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1aea23dcf3e0b02275ede97637ab172be1c6bfd2b9618d3aac9b2bc5f11acaa4/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:274 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~projected/kube-api-access-lm9zh DeviceMajor:0 DeviceMinor:341 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:348 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:274 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:279 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c2c3998f2f3e90dc73190e02bb652ae7a42c289d3d7c92a114eab460e226ccc/userdata/shm DeviceMajor:0 DeviceMinor:444 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2be4be0e9cfc3c8dedfe37691dab2da955e4cd5e14fc1a5cee7b9104f11586e0/userdata/shm DeviceMajor:0 DeviceMinor:446 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:274 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/26366d8c-76f1-479e-9715-6512f20bd3c6/volumes/kubernetes.io~projected/kube-api-access-ctgxg DeviceMajor:0 DeviceMinor:204 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:270 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~projected/kube-api-access-4sgm9 DeviceMajor:0 DeviceMinor:350 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ea837196-594f-42a6-981b-74717f55e790/volumes/kubernetes.io~projected/kube-api-access-qkns2 DeviceMajor:0 DeviceMinor:66 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c/volumes/kubernetes.io~projected/kube-api-access-qjpf6 DeviceMajor:0 DeviceMinor:67 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:335 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18595f28-dc19-4df0-a61b-905e22653608/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:340 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1baadc8-f14c-490f-9655-a55352e7eb8b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:65 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c478781b15d1eddbed88055f52b54ff23b89ea0b3c980619a6d32e795b310fc6/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dee8af8b6e9db4d10d99cdaf32ee9aa115336aa79d884d90bab5fe69811619ce/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:279 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98cd666dca399b17251fe1448ac855173ee885cc270031f12268dfb12b10bad9/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c6fe3b2750890436a0c7e02358d55145f299e80ba2ca1f043b8d75ac056398b/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:284 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbd8015f1ffb7ea0b632a028ed80b82cf572c9c7cc67514a94c4f2095bdd4053/userdata/shm DeviceMajor:0 DeviceMinor:442 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2c4e933-b200-47a6-94c8-1966aeb16884/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eda1485c80c9159bf77bb63e96a59ce816c02b7f3b84c283e36cdcb5595df434/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:347 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5baeb178-3616-4b1d-8515-d8ff71486666/volumes/kubernetes.io~projected/kube-api-access-qrbgq DeviceMajor:0 DeviceMinor:61 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:282 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:274 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:349 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d/volumes/kubernetes.io~projected/kube-api-access-6fhtq DeviceMajor:0 DeviceMinor:202 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:283 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46844a8e32c7b4cfa6f04a52ced4f17086e0311f0f1d2fa694589b7525f975b8/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2c4e933-b200-47a6-94c8-1966aeb16884/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:281 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c/volumes/kubernetes.io~projected/kube-api-access-9wfck DeviceMajor:0 DeviceMinor:62 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9ed2868-24ad-473c-bac4-01344c0e4bed/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/e8572e79-8cf8-4b15-b77f-ba409bd3b51f/volumes/kubernetes.io~projected/kube-api-access-c577v DeviceMajor:0 DeviceMinor:235 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~projected/kube-api-access-44ld4 DeviceMajor:0 DeviceMinor:305 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:303 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c034008e-a79f-4ebb-9316-d5d63bc1dbb9/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:346 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b75527010af335f6bc91af969b0d3f5a9ec9ae1caf1ec6b95d5c1224420b98f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1dd1e50-ecd0-4724-862d-752e9e1aecd0/volumes/kubernetes.io~projected/kube-api-access-k5dtj DeviceMajor:0 DeviceMinor:59 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8739003e-fba1-4d47-80e9-0f484d117b73/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:436 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d77255dd-93d5-4c30-8a4c-f4c7ea44d702/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:231 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/e8572e79-8cf8-4b15-b77f-ba409bd3b51f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:232 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:296 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a8667518758b9b4b88869f84365d55fc37b8e0deecc74895a4a13469dda1263/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:304 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:280 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:301 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9ed2868-24ad-473c-bac4-01344c0e4bed/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76a5a648-0585-408d-8556-c0e31dcea942/volumes/kubernetes.io~projected/kube-api-access-2hsxj DeviceMajor:0 DeviceMinor:57 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8572e79-8cf8-4b15-b77f-ba409bd3b51f/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:233 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9ed2868-24ad-473c-bac4-01344c0e4bed/volumes/kubernetes.io~projected/kube-api-access-c2r9f DeviceMajor:0 DeviceMinor:236 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1629c22a269b002d6230d7524a8ff25fa938d89393541ddd3513edd6ea73953a/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6962a1b2-43de-4da8-a1f4-af8101d42fd0/volumes/kubernetes.io~projected/kube-api-access-lsfjt DeviceMajor:0 DeviceMinor:58 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/4d408013-e118-42d1-809f-5d3b50f12565/volumes/kubernetes.io~projected/kube-api-access-j7sqc DeviceMajor:0 DeviceMinor:60 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2c4e933-b200-47a6-94c8-1966aeb16884/volumes/kubernetes.io~projected/kube-api-access-nt6gp DeviceMajor:0 DeviceMinor:63 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0/volumes/kubernetes.io~projected/kube-api-access-h5csc DeviceMajor:0 DeviceMinor:201 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ff711848f297a3eba2d1e200eb8c14f77a2569882c909d8515bb893f7830428/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1629c22a269b002 MacAddress:b2:f1:b7:38:05:52 Speed:10000 Mtu:8901} {Name:1a8667518758b9b MacAddress:72:44:e7:68:3d:37 Speed:10000 Mtu:8901} {Name:1aea23dcf3e0b02 MacAddress:ce:d6:28:12:71:f8 Speed:10000 Mtu:8901} {Name:1c2c3998f2f3e90 MacAddress:9a:a8:4f:ef:91:5b Speed:10000 Mtu:8901} {Name:2be4be0e9cfc3c8 MacAddress:46:e7:2d:89:a0:38 Speed:10000 Mtu:8901} {Name:46844a8e32c7b4c MacAddress:d2:e0:05:8a:b6:85 Speed:10000 Mtu:8901} {Name:74dba1cf8098e91 MacAddress:e2:3c:20:49:67:3a Speed:10000 Mtu:8901} {Name:8c6fe3b27508904 MacAddress:ce:3e:a8:56:21:31 Speed:10000 Mtu:8901} {Name:9ff711848f297a3 MacAddress:36:d5:23:b7:7e:16 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d1:ce:52:9d Speed:0 Mtu:9001} {Name:br-int MacAddress:36:38:46:ec:35:3f Speed:0 Mtu:8901} {Name:cbd8015f1ffb7ea MacAddress:5e:54:3e:66:71:d1 Speed:10000 Mtu:8901} {Name:dee8af8b6e9db4d MacAddress:7a:cc:f3:6e:ab:a8 Speed:10000 Mtu:8901} {Name:e04aadde8ef4790 MacAddress:76:eb:bb:4c:0a:e8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d1:ce:52:9d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:f2:f7:1e:32:7f:94 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:11:d2:e1:97:61 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 22:23:16.281819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.280859 6466 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 22:23:16.281819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.280906 6466 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 22:23:16.281819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.281300 6466 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 22:23:16.281819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.281331 6466 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-88.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 22:23:16.281819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.281547 6466 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 22:23:16.281819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.281560 6466 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 22:23:16.281819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.281579 6466 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:23:16.281819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.281597 6466 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:23:16.281819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.281734 6466 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 22:23:16.282595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.281964 6466 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 22:23:16.282595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.282122 6466 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 22:23:16.282595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.282173 6466 kubelet.go:493] "Attempting to sync node with API server" Nov 01 22:23:16.282595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.282188 6466 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 22:23:16.282595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.282203 6466 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 22:23:16.282595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.282216 6466 kubelet.go:405] "Adding apiserver pod source" Nov 01 22:23:16.282595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.282227 6466 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 22:23:16.283231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.283191 6466 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 22:23:16.283231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.283208 6466 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:23:16.283856 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.283834 6466 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 22:23:16.286584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.286570 6466 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 22:23:16.286584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.286584 6466 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:23:16.286685 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.286676 6466 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 22:23:16.286869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.286859 6466 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 22:23:16.286899 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.286888 6466 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 22:23:16.286982 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.286973 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 22:23:16.287021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.286988 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 22:23:16.287021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.286994 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 22:23:16.287021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.287000 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 22:23:16.287021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.287005 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 22:23:16.287021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.287011 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 22:23:16.287021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.287017 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 22:23:16.287021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.287023 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 22:23:16.287255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.287030 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 22:23:16.287255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.287036 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 22:23:16.287255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.287045 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 22:23:16.287255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.287054 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 22:23:16.287255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.287078 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 22:23:16.287255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.287084 6466 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 22:23:16.290759 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.290736 6466 server.go:1263] "Started kubelet" Nov 01 22:23:16.291712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.291147 6466 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 22:23:16.291712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.291430 6466 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 22:23:16.291712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.291516 6466 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 22:23:16.291460 ip-10-0-134-88 systemd[1]: Started Kubernetes Kubelet. Nov 01 22:23:16.294134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.293262 6466 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 22:23:16.296250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.296233 6466 server.go:317] "Adding debug handlers to kubelet server" Nov 01 22:23:16.298707 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.298660 6466 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 22:23:16.298707 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.298682 6466 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 22:23:16.298855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.298725 6466 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:05 +0000 UTC" deadline="2028-08-15 06:25:18.355757404 +0000 UTC" Nov 01 22:23:16.298855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.298742 6466 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15656h2m2.057018359s" Nov 01 22:23:16.298855 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:16.298773 6466 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-88.us-east-2.compute.internal\" not found" Nov 01 22:23:16.299161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.299139 6466 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 22:23:16.299484 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.299462 6466 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 22:23:16.299484 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.299484 6466 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 22:23:16.299636 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.299554 6466 factory.go:55] Registering systemd factory Nov 01 22:23:16.299636 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.299575 6466 factory.go:223] Registration of the systemd container factory successfully Nov 01 22:23:16.300016 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.299851 6466 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:16.301382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.301307 6466 factory.go:153] Registering CRI-O factory Nov 01 22:23:16.301382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.301328 6466 factory.go:223] Registration of the crio container factory successfully Nov 01 22:23:16.301382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.301377 6466 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 22:23:16.301565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.301398 6466 factory.go:103] Registering Raw factory Nov 01 22:23:16.301565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.301414 6466 manager.go:1199] Started watching for new ooms in manager Nov 01 22:23:16.301750 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.301736 6466 manager.go:319] Starting recovery of all containers Nov 01 22:23:16.302701 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.302681 6466 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:16.302986 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.302947 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18595f28-dc19-4df0-a61b-905e22653608" volumeName="kubernetes.io/projected/18595f28-dc19-4df0-a61b-905e22653608-kube-api-access-lm9zh" seLinuxMountContext="" Nov 01 22:23:16.303051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.302992 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d408013-e118-42d1-809f-5d3b50f12565" volumeName="kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-binary-copy" seLinuxMountContext="" Nov 01 22:23:16.303051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.303015 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c034008e-a79f-4ebb-9316-d5d63bc1dbb9" volumeName="kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 01 22:23:16.303051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.303029 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d77255dd-93d5-4c30-8a4c-f4c7ea44d702" volumeName="kubernetes.io/empty-dir/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-textfile" seLinuxMountContext="" Nov 01 22:23:16.303980 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.303960 6466 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.304945 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.304979 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d" volumeName="kubernetes.io/projected/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-kube-api-access-6fhtq" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305001 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55" volumeName="kubernetes.io/configmap/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-metrics-client-ca" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305016 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c" volumeName="kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-config" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305100 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c034008e-a79f-4ebb-9316-d5d63bc1dbb9" volumeName="kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305130 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d77255dd-93d5-4c30-8a4c-f4c7ea44d702" volumeName="kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-metrics-client-ca" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305143 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9ed2868-24ad-473c-bac4-01344c0e4bed" volumeName="kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-tls" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305161 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76a5a648-0585-408d-8556-c0e31dcea942" volumeName="kubernetes.io/projected/76a5a648-0585-408d-8556-c0e31dcea942-kube-api-access-2hsxj" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305174 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55" volumeName="kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305192 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55" volumeName="kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-grpc-tls" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305210 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c" volumeName="kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-env-overrides" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305246 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c" volumeName="kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-script-lib" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305258 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1baadc8-f14c-490f-9655-a55352e7eb8b" volumeName="kubernetes.io/secret/e1baadc8-f14c-490f-9655-a55352e7eb8b-agent-certs" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305271 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9ed2868-24ad-473c-bac4-01344c0e4bed" volumeName="kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-metrics-client-ca" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305290 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18595f28-dc19-4df0-a61b-905e22653608" volumeName="kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-tls" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305323 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d408013-e118-42d1-809f-5d3b50f12565" volumeName="kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305343 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55" volumeName="kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305356 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8739003e-fba1-4d47-80e9-0f484d117b73" volumeName="kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305373 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0" volumeName="kubernetes.io/projected/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-api-access-h5csc" seLinuxMountContext="" Nov 01 22:23:16.305379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305389 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1baadc8-f14c-490f-9655-a55352e7eb8b" volumeName="kubernetes.io/configmap/e1baadc8-f14c-490f-9655-a55352e7eb8b-konnectivity-ca" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305403 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea837196-594f-42a6-981b-74717f55e790" volumeName="kubernetes.io/projected/ea837196-594f-42a6-981b-74717f55e790-kube-api-access-qkns2" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305420 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305432 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55" volumeName="kubernetes.io/empty-dir/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-tmp" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305492 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18595f28-dc19-4df0-a61b-905e22653608" volumeName="kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-metrics-server-audit-profiles" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305519 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26366d8c-76f1-479e-9715-6512f20bd3c6" volumeName="kubernetes.io/empty-dir/26366d8c-76f1-479e-9715-6512f20bd3c6-tmp-dir" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305537 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5baeb178-3616-4b1d-8515-d8ff71486666" volumeName="kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-multus-daemon-config" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305553 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9ed2868-24ad-473c-bac4-01344c0e4bed" volumeName="kubernetes.io/empty-dir/e9ed2868-24ad-473c-bac4-01344c0e4bed-volume-directive-shadow" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305575 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-cluster-tls-config" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305591 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26366d8c-76f1-479e-9715-6512f20bd3c6" volumeName="kubernetes.io/configmap/26366d8c-76f1-479e-9715-6512f20bd3c6-config-volume" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305608 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26366d8c-76f1-479e-9715-6512f20bd3c6" volumeName="kubernetes.io/secret/26366d8c-76f1-479e-9715-6512f20bd3c6-metrics-tls" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305690 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55" volumeName="kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-tls" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305706 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c034008e-a79f-4ebb-9316-d5d63bc1dbb9" volumeName="kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client" seLinuxMountContext="" Nov 01 22:23:16.306185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.305718 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d77255dd-93d5-4c30-8a4c-f4c7ea44d702" volumeName="kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306201 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8572e79-8cf8-4b15-b77f-ba409bd3b51f" volumeName="kubernetes.io/projected/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-kube-api-access-c577v" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306240 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9ed2868-24ad-473c-bac4-01344c0e4bed" volumeName="kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306251 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-tls-assets" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306258 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dbe2700-733d-4ebd-9f47-7b9f30bb62fd" volumeName="kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306267 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55" volumeName="kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306274 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1dd1e50-ecd0-4724-862d-752e9e1aecd0" volumeName="kubernetes.io/projected/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-kube-api-access-k5dtj" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306283 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-web-config" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306290 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306298 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c034008e-a79f-4ebb-9316-d5d63bc1dbb9" volumeName="kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-serving-certs-ca-bundle" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306309 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c034008e-a79f-4ebb-9316-d5d63bc1dbb9" volumeName="kubernetes.io/projected/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-kube-api-access-4sgm9" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306319 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c034008e-a79f-4ebb-9316-d5d63bc1dbb9" volumeName="kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-federate-client-tls" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306327 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0" volumeName="kubernetes.io/secret/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306348 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d77255dd-93d5-4c30-8a4c-f4c7ea44d702" volumeName="kubernetes.io/projected/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-kube-api-access-bntx6" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306355 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55" volumeName="kubernetes.io/projected/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-kube-api-access-44ld4" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306362 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d408013-e118-42d1-809f-5d3b50f12565" volumeName="kubernetes.io/projected/4d408013-e118-42d1-809f-5d3b50f12565-kube-api-access-j7sqc" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306371 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5baeb178-3616-4b1d-8515-d8ff71486666" volumeName="kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-cni-binary-copy" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306378 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6962a1b2-43de-4da8-a1f4-af8101d42fd0" volumeName="kubernetes.io/configmap/6962a1b2-43de-4da8-a1f4-af8101d42fd0-serviceca" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306385 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6962a1b2-43de-4da8-a1f4-af8101d42fd0" volumeName="kubernetes.io/projected/6962a1b2-43de-4da8-a1f4-af8101d42fd0-kube-api-access-lsfjt" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306395 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8572e79-8cf8-4b15-b77f-ba409bd3b51f" volumeName="kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-tls" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306409 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8572e79-8cf8-4b15-b77f-ba409bd3b51f" volumeName="kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306417 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d408013-e118-42d1-809f-5d3b50f12565" volumeName="kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306427 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55" volumeName="kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306441 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c034008e-a79f-4ebb-9316-d5d63bc1dbb9" volumeName="kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-metrics-client-ca" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306448 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1dd1e50-ecd0-4724-862d-752e9e1aecd0" volumeName="kubernetes.io/configmap/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-iptables-alerter-script" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306456 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-out" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306463 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18595f28-dc19-4df0-a61b-905e22653608" volumeName="kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306470 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" volumeName="kubernetes.io/projected/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-kube-api-access-9wfck" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306477 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c" volumeName="kubernetes.io/secret/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306485 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d77255dd-93d5-4c30-8a4c-f4c7ea44d702" volumeName="kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-tls" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306492 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9ed2868-24ad-473c-bac4-01344c0e4bed" volumeName="kubernetes.io/projected/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-api-access-c2r9f" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306501 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-kube-api-access-78lnx" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306509 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18595f28-dc19-4df0-a61b-905e22653608" volumeName="kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-client-ca-bundle" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306518 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26366d8c-76f1-479e-9715-6512f20bd3c6" volumeName="kubernetes.io/projected/26366d8c-76f1-479e-9715-6512f20bd3c6-kube-api-access-ctgxg" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306526 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c" volumeName="kubernetes.io/projected/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-kube-api-access-qjpf6" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306534 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d77255dd-93d5-4c30-8a4c-f4c7ea44d702" volumeName="kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306541 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c4e933-b200-47a6-94c8-1966aeb16884" volumeName="kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-tmp" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306549 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c4e933-b200-47a6-94c8-1966aeb16884" volumeName="kubernetes.io/projected/f2c4e933-b200-47a6-94c8-1966aeb16884-kube-api-access-nt6gp" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306557 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5baeb178-3616-4b1d-8515-d8ff71486666" volumeName="kubernetes.io/projected/5baeb178-3616-4b1d-8515-d8ff71486666-kube-api-access-qrbgq" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306567 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-volume" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306574 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2c4e933-b200-47a6-94c8-1966aeb16884" volumeName="kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-tuned" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306581 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18595f28-dc19-4df0-a61b-905e22653608" volumeName="kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306589 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d" volumeName="kubernetes.io/secret/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-cert" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306597 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c034008e-a79f-4ebb-9316-d5d63bc1dbb9" volumeName="kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-client-tls" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306604 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0" volumeName="kubernetes.io/empty-dir/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-data-volume" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306610 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8572e79-8cf8-4b15-b77f-ba409bd3b51f" volumeName="kubernetes.io/configmap/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-metrics-client-ca" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306618 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9ed2868-24ad-473c-bac4-01344c0e4bed" volumeName="kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306627 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76a5a648-0585-408d-8556-c0e31dcea942" volumeName="kubernetes.io/empty-dir/76a5a648-0585-408d-8556-c0e31dcea942-tmp-dir" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306636 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21a7a1c-6db6-422d-a24f-5bdc6b4a700c" volumeName="kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306644 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306652 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18595f28-dc19-4df0-a61b-905e22653608" volumeName="kubernetes.io/empty-dir/18595f28-dc19-4df0-a61b-905e22653608-audit-log" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306660 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0" volumeName="kubernetes.io/configmap/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306668 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-metrics-client-ca" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306674 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-main-db" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306681 6466 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" volumeName="kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306689 6466 reconstruct.go:97] "Volume reconstruction finished" Nov 01 22:23:16.306792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.306695 6466 reconciler.go:29] "Reconciler: start to sync state" Nov 01 22:23:16.309812 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.309797 6466 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 22:23:16.331212 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.331177 6466 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 22:23:16.332760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.332733 6466 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 22:23:16.332760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.332758 6466 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 22:23:16.332910 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.332781 6466 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 22:23:16.332910 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:16.332821 6466 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 22:23:16.336336 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.336317 6466 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:16.354913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.354887 6466 generic.go:356] "Generic (PLEG): container finished" podID="d77255dd-93d5-4c30-8a4c-f4c7ea44d702" containerID="8c95e5f56b5cc01888ef445435aa21a5ba1acaa7343cb3eae07fb88686439a45" exitCode=0 Nov 01 22:23:16.370790 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.370744 6466 generic.go:356] "Generic (PLEG): container finished" podID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerID="d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8" exitCode=0 Nov 01 22:23:16.378318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.378283 6466 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="4e8966d105cd3ab3ce28405fca93644ce16b9675f66cc6b20c8cb5f55d431e46" exitCode=0 Nov 01 22:23:16.378318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.378317 6466 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="ff32440f3b9dc4bb22abeea5de0926a86257971dca235095624a8a96e7c791ad" exitCode=0 Nov 01 22:23:16.378504 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.378326 6466 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="6b718c65ea4aec999d784c5c7d5a7ea645e6c6088b7ecbac5caa2dc00086212c" exitCode=0 Nov 01 22:23:16.378504 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.378334 6466 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="340dc9ed9923b69c7bb9303140fced897a53be352d54f07c1dd57eb200cff9a7" exitCode=0 Nov 01 22:23:16.378504 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.378340 6466 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="3d70c2e948fd114135ed186142d485e86353941db92d968ad2146aa3779db9f4" exitCode=0 Nov 01 22:23:16.378504 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.378348 6466 generic.go:356] "Generic (PLEG): container finished" podID="4d408013-e118-42d1-809f-5d3b50f12565" containerID="81c9e2fc629edb4a77c67bdaf27e4b7b66d3379d9fb62381d43ecfc877287202" exitCode=0 Nov 01 22:23:16.381733 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.381705 6466 generic.go:356] "Generic (PLEG): container finished" podID="615e11a436a671e663c9b9217b7e9606" containerID="edcec4ee6b53796f73a7d1705c13b0dac0aad245a6d297e20a1de6571a358aef" exitCode=0 Nov 01 22:23:16.413190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.413134 6466 manager.go:324] Recovery completed Nov 01 22:23:16.433004 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:16.432973 6466 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 22:23:16.438008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.437986 6466 cpu_manager.go:226] "Starting" policy="none" Nov 01 22:23:16.438008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438006 6466 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 22:23:16.438156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438028 6466 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 22:23:16.438234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438216 6466 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 22:23:16.438266 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438236 6466 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 22:23:16.438266 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438260 6466 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 22:23:16.438322 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438269 6466 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 22:23:16.438322 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438276 6466 policy_none.go:50] "Start" Nov 01 22:23:16.438322 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438290 6466 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 22:23:16.438322 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438304 6466 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 22:23:16.438455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438445 6466 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 22:23:16.438485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438456 6466 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 22:23:16.438485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.438462 6466 policy_none.go:44] "Start" Nov 01 22:23:16.443065 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.443039 6466 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 22:23:16.446189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.446177 6466 manager.go:341] "Starting Device Plugin manager" Nov 01 22:23:16.446884 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:16.446206 6466 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 22:23:16.446953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.446892 6466 server.go:87] "Starting device plugin registration server" Nov 01 22:23:16.447238 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.447225 6466 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 22:23:16.447285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.447240 6466 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 22:23:16.447367 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.447346 6466 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 22:23:16.447519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.447475 6466 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:23:16.447519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.447521 6466 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 22:23:16.447519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.447526 6466 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 22:23:16.447736 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.447590 6466 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:23:16.447510385Z","UUID":"15b24a8c-caac-45da-ad6b-ff447ee0cd8d","Handler":null,"Name":"","Endpoint":""} Nov 01 22:23:16.449017 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:16.448994 6466 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 22:23:16.449491 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.449474 6466 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 22:23:16.449575 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.449512 6466 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:23:16.547966 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.547927 6466 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:23:16.550169 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.550145 6466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:23:16.550322 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.550179 6466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:23:16.550322 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.550191 6466 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-88.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:23:16.550322 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.550241 6466 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.561582 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.561559 6466 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.561683 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.561640 6466 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.633752 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.633670 6466 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal"] Nov 01 22:23:16.634475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.634454 6466 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.634542 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.634505 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" event={"ID":"615e11a436a671e663c9b9217b7e9606","Type":"ContainerStarted","Data":"74c446ffefd716924f6845cc5582c75daefa293c851c182a8fcf98ee4c0a4ce4"} Nov 01 22:23:16.634579 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.634556 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" event={"ID":"615e11a436a671e663c9b9217b7e9606","Type":"ContainerDied","Data":"edcec4ee6b53796f73a7d1705c13b0dac0aad245a6d297e20a1de6571a358aef"} Nov 01 22:23:16.634632 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.634576 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" event={"ID":"615e11a436a671e663c9b9217b7e9606","Type":"ContainerStarted","Data":"df30684be22fdfc276dbba87c25d24c6e68907c5da7ff9184d651db76991a3f1"} Nov 01 22:23:16.634632 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.634610 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" event={"ID":"da9c6d0ee7daac240eb44f117e691eaf","Type":"ContainerStarted","Data":"83b103e1ff219a7da8deb08d0f030f273cbdc6ce5dc4714220fbbb799f294a3e"} Nov 01 22:23:16.634632 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.634628 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" event={"ID":"da9c6d0ee7daac240eb44f117e691eaf","Type":"ContainerStarted","Data":"5b75527010af335f6bc91af969b0d3f5a9ec9ae1caf1ec6b95d5c1224420b98f"} Nov 01 22:23:16.634763 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.634752 6466 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.646159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.646139 6466 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:16.646242 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:16.646179 6466 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.646484 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.646468 6466 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:16.646520 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:16.646508 6466 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.711886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.711824 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9c6d0ee7daac240eb44f117e691eaf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"da9c6d0ee7daac240eb44f117e691eaf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.711886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.711858 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.711886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.711874 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.812483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.812448 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.812483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.812404 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.812655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.812524 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.812655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.812544 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9c6d0ee7daac240eb44f117e691eaf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"da9c6d0ee7daac240eb44f117e691eaf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.812655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.812557 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/615e11a436a671e663c9b9217b7e9606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"615e11a436a671e663c9b9217b7e9606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:16.812655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:16.812587 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da9c6d0ee7daac240eb44f117e691eaf-config\") pod \"kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal\" (UID: \"da9c6d0ee7daac240eb44f117e691eaf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:17.282439 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.282403 6466 apiserver.go:52] "Watching apiserver" Nov 01 22:23:17.296691 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.296660 6466 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 22:23:17.297331 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.297310 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal","openshift-dns/dns-default-sh7c7","openshift-monitoring/node-exporter-cw7qc","openshift-multus/network-metrics-daemon-8cbk8","kube-system/konnectivity-agent-fxlz9","openshift-dns/node-resolver-mfnxk","openshift-cluster-node-tuning-operator/tuned-f4rmp","openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz","openshift-image-registry/node-ca-f9snb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal","openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr","openshift-multus/multus-additional-cni-plugins-t8rz8","kube-system/global-pull-secret-syncer-cg4rv","openshift-network-diagnostics/network-check-target-vzk9n","openshift-monitoring/thanos-querier-675964fd8b-v45n9","openshift-network-operator/iptables-alerter-v2bl8","openshift-ovn-kubernetes/ovnkube-node-zntdq","openshift-ingress-canary/ingress-canary-77bdk","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-qj4tr","openshift-insights/insights-runtime-extractor-kc74w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt","openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv","openshift-monitoring/metrics-server-796bf69784-66sjq"] Nov 01 22:23:17.302201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.302165 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 22:23:17.302363 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.302347 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 22:23:17.303121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.303080 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:17.303243 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.303226 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:17.303332 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.303307 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vv5lb\"" Nov 01 22:23:17.303394 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.303375 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fpxpg\"" Nov 01 22:23:17.303558 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.303537 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dclh4\"" Nov 01 22:23:17.303654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.303640 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 22:23:17.303703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.303656 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:17.304858 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.304839 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 22:23:17.305369 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.305352 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 22:23:17.305551 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.305531 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9pl6q\"" Nov 01 22:23:17.305639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.305612 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:17.306257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.306231 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 22:23:17.307518 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.307498 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 22:23:17.307600 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.307516 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 22:23:17.307600 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.307516 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 22:23:17.307600 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.307572 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p26bd\"" Nov 01 22:23:17.307751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.307654 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 22:23:17.307859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.307841 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-79k59\"" Nov 01 22:23:17.308124 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.308098 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 22:23:17.308250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.308233 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 22:23:17.308875 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.308853 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4hpp8\"" Nov 01 22:23:17.308934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.308864 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qjk9v\"" Nov 01 22:23:17.309318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.309298 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zwl5x\"" Nov 01 22:23:17.309499 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.309486 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j5zd8\"" Nov 01 22:23:17.309565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.309532 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 22:23:17.309742 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.309726 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 22:23:17.310354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.310337 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 22:23:17.310766 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.310571 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:17.310848 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.310819 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vvcmq\"" Nov 01 22:23:17.310848 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.310827 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5scqq\"" Nov 01 22:23:17.311044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.311027 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 22:23:17.311091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.311057 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 22:23:17.311918 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.311893 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 22:23:17.311997 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.311944 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 22:23:17.311997 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.311949 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:23:17.312386 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.312367 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 22:23:17.312471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.312400 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 22:23:17.312471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.312412 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 22:23:17.312471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.312425 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 22:23:17.312617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.312476 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4kglz\"" Nov 01 22:23:17.312617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.312534 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 22:23:17.312617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.312535 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 22:23:17.312884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.312868 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 22:23:17.312975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.312932 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4jwn7\"" Nov 01 22:23:17.313035 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.313004 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:17.313094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.313079 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:23:17.313265 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.313251 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 22:23:17.313337 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.313290 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4bt99\"" Nov 01 22:23:17.313337 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.313331 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 22:23:17.313449 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.313358 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 22:23:17.313998 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.313970 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 22:23:17.313998 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.313972 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 22:23:17.314138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314013 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 22:23:17.314138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314024 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 22:23:17.314138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314068 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 22:23:17.314250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314144 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 22:23:17.314250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314208 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:23:17.314250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314222 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 22:23:17.314250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314236 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-kubernetes\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.314458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314309 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-conf\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.314458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314392 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-socket-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.314458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314431 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-modprobe-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.314564 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314459 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-run\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.314564 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314475 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-host\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.314564 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314505 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wfck\" (UniqueName: \"kubernetes.io/projected/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-kube-api-access-9wfck\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:23:17.314564 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314541 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8739003e-fba1-4d47-80e9-0f484d117b73-original-pull-secret\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:23:17.314724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314544 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-lib-modules\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.314724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314687 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6gp\" (UniqueName: \"kubernetes.io/projected/f2c4e933-b200-47a6-94c8-1966aeb16884-kube-api-access-nt6gp\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.314813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314716 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.314813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314758 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-sys-fs\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.314813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314785 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 22:23:17.314813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314807 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-kubelet-config\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:23:17.314991 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314792 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 22:23:17.314991 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314879 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 22:23:17.314991 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314882 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-n7fmc\"" Nov 01 22:23:17.315271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315253 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zj5nt\"" Nov 01 22:23:17.315346 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315311 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 22:23:17.315402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315371 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 22:23:17.315402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.314845 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-sys\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.315494 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315424 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-t7pjg\"" Nov 01 22:23:17.315494 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315419 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:23:17.315494 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315442 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 22:23:17.315494 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315473 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-systemd\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.315636 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315504 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 22:23:17.315636 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315505 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-dbus\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315513 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315916 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.315933 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-metrics-certs\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.316117 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.316169 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-var-lib-kubelet\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.316229 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-registration-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.316266 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysconfig\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.316363 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-tuned\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.316398 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-tmp\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.316429 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkns2\" (UniqueName: \"kubernetes.io/projected/ea837196-594f-42a6-981b-74717f55e790-kube-api-access-qkns2\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.316459 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-device-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.316487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.316485 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-etc-selinux\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.317129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.316685 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-tuned\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.317129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.316791 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c4e933-b200-47a6-94c8-1966aeb16884-tmp\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.317350 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.317326 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 22:23:17.317698 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.317641 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 22:23:17.318094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.317706 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 22:23:17.318819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.318799 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mn4xg\"" Nov 01 22:23:17.320358 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.320340 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 22:23:17.320657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.320636 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 22:23:17.320756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.320671 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 22:23:17.320756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.320685 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 22:23:17.320842 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.320787 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7b27s7rg7h8jo\"" Nov 01 22:23:17.320886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.320854 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-w9c7x\"" Nov 01 22:23:17.320942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.320917 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 22:23:17.321522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.321503 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c91l7l24aggst\"" Nov 01 22:23:17.322056 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.322038 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wfck\" (UniqueName: \"kubernetes.io/projected/b21a7a1c-6db6-422d-a24f-5bdc6b4a700c-kube-api-access-9wfck\") pod \"network-metrics-daemon-8cbk8\" (UID: \"b21a7a1c-6db6-422d-a24f-5bdc6b4a700c\") " pod="openshift-multus/network-metrics-daemon-8cbk8" Nov 01 22:23:17.322154 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.322139 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6gp\" (UniqueName: \"kubernetes.io/projected/f2c4e933-b200-47a6-94c8-1966aeb16884-kube-api-access-nt6gp\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.323121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.323089 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkns2\" (UniqueName: \"kubernetes.io/projected/ea837196-594f-42a6-981b-74717f55e790-kube-api-access-qkns2\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.396729 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.396696 6466 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:17.396871 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.396762 6466 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:17.401281 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.401259 6466 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 22:23:17.402188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402169 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 22:23:17.402262 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402187 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8x5xq\"" Nov 01 22:23:17.402311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402287 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 22:23:17.402360 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402331 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 22:23:17.402360 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402334 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 22:23:17.402483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402464 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 22:23:17.402531 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402497 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 22:23:17.402531 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402501 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 22:23:17.402624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402533 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 22:23:17.402624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402542 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 22:23:17.402624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402560 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 22:23:17.402624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.402577 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 22:23:17.403430 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.403407 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8zdb7\"" Nov 01 22:23:17.403430 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.403410 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 22:23:17.403782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.403765 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 22:23:17.406837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.406811 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 22:23:17.409422 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.409402 6466 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:17.409506 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:17.409457 6466 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:17.409799 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.409782 6466 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:17.409909 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:17.409815 6466 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-88.us-east-2.compute.internal" Nov 01 22:23:17.417422 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417396 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-out\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.417591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417424 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.417591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417442 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26366d8c-76f1-479e-9715-6512f20bd3c6-metrics-tls\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:23:17.417591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417465 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-socket-dir-parent\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.417591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417541 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-host\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.417591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417546 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-out\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.417591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417570 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.417833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417657 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-host\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.417833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417666 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.417833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417688 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-netd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.417833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417718 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.417833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417742 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26366d8c-76f1-479e-9715-6512f20bd3c6-metrics-tls\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:23:17.417833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417745 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-tls\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.417833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417792 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e9ed2868-24ad-473c-bac4-01344c0e4bed-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.417833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417817 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-lib-modules\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.418224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417883 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-textfile\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.418224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417917 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntx6\" (UniqueName: \"kubernetes.io/projected/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-kube-api-access-bntx6\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.418224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417932 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e9ed2868-24ad-473c-bac4-01344c0e4bed-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.418224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.417949 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-tls\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.418224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418007 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-tls\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.418224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418038 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-textfile\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.418224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418090 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:23:17.418224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418102 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-lib-modules\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.418224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418142 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1baadc8-f14c-490f-9655-a55352e7eb8b-agent-certs\") pod \"konnectivity-agent-fxlz9\" (UID: \"e1baadc8-f14c-490f-9655-a55352e7eb8b\") " pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:23:17.418224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418173 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-multus\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.418224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418215 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-multus-daemon-config\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.418693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418241 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-accelerators-collector-config\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.418693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418285 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-metrics-client-ca\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.418693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418312 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6962a1b2-43de-4da8-a1f4-af8101d42fd0-host\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:23:17.418693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418353 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:23:17.418693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418338 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-registration-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.418693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418402 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-conf-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.418693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418444 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysconfig\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.419027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418839 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-systemd-units\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.419027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418886 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-config\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.419027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.418922 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjpf6\" (UniqueName: \"kubernetes.io/projected/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-kube-api-access-qjpf6\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.420298 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.420237 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-multus-daemon-config\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.420298 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.420262 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-tls\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.420471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.420363 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.420471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.420377 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1baadc8-f14c-490f-9655-a55352e7eb8b-agent-certs\") pod \"konnectivity-agent-fxlz9\" (UID: \"e1baadc8-f14c-490f-9655-a55352e7eb8b\") " pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:23:17.423881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.423395 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9zh\" (UniqueName: \"kubernetes.io/projected/18595f28-dc19-4df0-a61b-905e22653608-kube-api-access-lm9zh\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.423881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.423497 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:23:17.423881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.423694 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a5a648-0585-408d-8556-c0e31dcea942-tmp-dir\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:23:17.423881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.423741 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-registration-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.423881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.423765 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-metrics-client-ca\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.423881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.423747 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-sys\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.423881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.423783 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-accelerators-collector-config\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.423881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.423841 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a5a648-0585-408d-8556-c0e31dcea942-tmp-dir\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:23:17.423881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.423854 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-metrics-client-ca\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.424397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424033 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:23:17.424397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424093 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/18595f28-dc19-4df0-a61b-905e22653608-audit-log\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.424397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424097 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysconfig\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.424397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424152 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrbgq\" (UniqueName: \"kubernetes.io/projected/5baeb178-3616-4b1d-8515-d8ff71486666-kube-api-access-qrbgq\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.424397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424200 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.424397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424251 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bntx6\" (UniqueName: \"kubernetes.io/projected/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-kube-api-access-bntx6\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.424397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424261 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.424397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424289 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-config\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.424782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424449 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.424782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424464 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-metrics-client-ca\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.424782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424600 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/18595f28-dc19-4df0-a61b-905e22653608-audit-log\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.424782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424657 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-kubernetes\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.424782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424683 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.425062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424806 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.425062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424834 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:23:17.425062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424880 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.425062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.424884 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-run\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.425062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425036 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:23:17.425062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425058 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425073 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-cnibin\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425099 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-kubernetes\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425099 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-sys-fs\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425197 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425227 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-kubelet\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425246 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-etc-kubernetes\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425268 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-slash\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425288 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425319 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-sys-fs\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425353 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425359 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-run\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425374 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.425406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425397 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78lnx\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-kube-api-access-78lnx\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.426030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425654 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7sqc\" (UniqueName: \"kubernetes.io/projected/4d408013-e118-42d1-809f-5d3b50f12565-kube-api-access-j7sqc\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.426030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425678 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-iptables-alerter-script\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:23:17.426030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425714 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.426030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425723 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-dbus\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:23:17.426030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425744 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-etc-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.426030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425766 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.426030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425929 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.426030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425980 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-web-config\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.426030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.425977 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.426030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426003 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ld4\" (UniqueName: \"kubernetes.io/projected/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-kube-api-access-44ld4\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.426030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426023 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-dbus\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426053 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-kubelet\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426071 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-log-socket\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426143 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovn-node-metrics-cert\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426185 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-device-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426186 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-iptables-alerter-script\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426228 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-var-lib-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426284 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-env-overrides\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426315 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-tls\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426346 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426370 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426380 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovn-node-metrics-cert\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426397 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-env-overrides\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426450 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1baadc8-f14c-490f-9655-a55352e7eb8b-konnectivity-ca\") pod \"konnectivity-agent-fxlz9\" (UID: \"e1baadc8-f14c-490f-9655-a55352e7eb8b\") " pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426490 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-tmp\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426510 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-system-cni-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426530 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426551 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-cert\") pod \"ingress-canary-77bdk\" (UID: \"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d\") " pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426571 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.426598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426573 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5csc\" (UniqueName: \"kubernetes.io/projected/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-api-access-h5csc\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426655 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-tmp\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426730 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426780 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-web-config\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426802 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-device-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426805 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426878 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-netns\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426918 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.426948 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-cert\") pod \"ingress-canary-77bdk\" (UID: \"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d\") " pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427007 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427057 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-systemd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427068 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-tls\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427149 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-script-lib\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427214 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427232 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1baadc8-f14c-490f-9655-a55352e7eb8b-konnectivity-ca\") pod \"konnectivity-agent-fxlz9\" (UID: \"e1baadc8-f14c-490f-9655-a55352e7eb8b\") " pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427256 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427271 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2r9f\" (UniqueName: \"kubernetes.io/projected/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-api-access-c2r9f\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427315 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427406 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76a5a648-0585-408d-8556-c0e31dcea942-hosts-file\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427427 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-ovnkube-script-lib\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.427506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427482 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-volume\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427521 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427531 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427556 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427587 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427668 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctgxg\" (UniqueName: \"kubernetes.io/projected/26366d8c-76f1-479e-9715-6512f20bd3c6-kube-api-access-ctgxg\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427679 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427706 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-volume\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427707 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-bin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427823 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-sys\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427857 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427857 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-sys\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.427893 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428019 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428025 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428062 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-data-volume\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428098 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-k8s-cni-cncf-io\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428185 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-data-volume\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428248 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428268 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428287 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-bin\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.428618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428307 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsfjt\" (UniqueName: \"kubernetes.io/projected/6962a1b2-43de-4da8-a1f4-af8101d42fd0-kube-api-access-lsfjt\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428672 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428783 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-grpc-tls\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428989 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-grpc-tls\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429021 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26366d8c-76f1-479e-9715-6512f20bd3c6-config-volume\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429042 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429057 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-var-lib-kubelet\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.428702 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429076 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-ovn\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429144 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6962a1b2-43de-4da8-a1f4-af8101d42fd0-serviceca\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429202 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-binary-copy\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429261 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-var-lib-kubelet\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429261 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-system-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429606 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.429657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429635 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-cni-binary-copy\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429670 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-hostroot\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429695 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26366d8c-76f1-479e-9715-6512f20bd3c6-config-volume\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429705 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429725 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-binary-copy\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429734 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-client-ca-bundle\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429771 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26366d8c-76f1-479e-9715-6512f20bd3c6-tmp-dir\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429787 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5baeb178-3616-4b1d-8515-d8ff71486666-cni-binary-copy\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429800 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-conf\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429856 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26366d8c-76f1-479e-9715-6512f20bd3c6-tmp-dir\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429918 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgm9\" (UniqueName: \"kubernetes.io/projected/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-kube-api-access-4sgm9\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429928 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-sysctl-conf\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429918 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.429966 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-client-ca-bundle\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430041 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fhtq\" (UniqueName: \"kubernetes.io/projected/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-kube-api-access-6fhtq\") pod \"ingress-canary-77bdk\" (UID: \"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d\") " pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430068 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-modprobe-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430093 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430135 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430160 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-netns\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430182 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430207 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-federate-client-tls\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430211 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-modprobe-d\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430217 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6962a1b2-43de-4da8-a1f4-af8101d42fd0-serviceca\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430230 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-crio-socket\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430252 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-cnibin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430298 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-kubelet-config\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430343 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-metrics-server-audit-profiles\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430369 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-systemd\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.430383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430397 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-client-tls\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430424 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-client-certs\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430461 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430491 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hsxj\" (UniqueName: \"kubernetes.io/projected/76a5a648-0585-408d-8556-c0e31dcea942-kube-api-access-2hsxj\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430490 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430519 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430565 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-serving-certs-ca-bundle\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430573 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430594 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-host-slash\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430627 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c577v\" (UniqueName: \"kubernetes.io/projected/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-kube-api-access-c577v\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430654 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-root\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430680 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-multus-certs\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430705 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-etc-selinux\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430732 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-node-log\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430760 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-metrics-client-ca\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430772 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/18595f28-dc19-4df0-a61b-905e22653608-secret-metrics-server-client-certs\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430787 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430819 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430829 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430834 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2c4e933-b200-47a6-94c8-1966aeb16884-etc-systemd\") pod \"tuned-f4rmp\" (UID: \"f2c4e933-b200-47a6-94c8-1966aeb16884\") " pod="openshift-cluster-node-tuning-operator/tuned-f4rmp" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430868 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5dtj\" (UniqueName: \"kubernetes.io/projected/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-kube-api-access-k5dtj\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430894 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-socket-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430920 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-wtmp\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430993 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430759 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjpf6\" (UniqueName: \"kubernetes.io/projected/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-kube-api-access-qjpf6\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431033 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-federate-client-tls\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431045 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-telemeter-client-tls\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.430998 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9zh\" (UniqueName: \"kubernetes.io/projected/18595f28-dc19-4df0-a61b-905e22653608-kube-api-access-lm9zh\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431122 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-socket-dir\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431099 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea837196-594f-42a6-981b-74717f55e790-etc-selinux\") pod \"aws-ebs-csi-driver-node-kv4rt\" (UID: \"ea837196-594f-42a6-981b-74717f55e790\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kv4rt" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431129 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8739003e-fba1-4d47-80e9-0f484d117b73-kubelet-config\") pod \"global-pull-secret-syncer-cg4rv\" (UID: \"8739003e-fba1-4d47-80e9-0f484d117b73\") " pod="kube-system/global-pull-secret-syncer-cg4rv" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431243 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d408013-e118-42d1-809f-5d3b50f12565-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431260 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-serving-certs-ca-bundle\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431318 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431372 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrbgq\" (UniqueName: \"kubernetes.io/projected/5baeb178-3616-4b1d-8515-d8ff71486666-kube-api-access-qrbgq\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431416 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431428 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-metrics-client-ca\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.431830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431492 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/18595f28-dc19-4df0-a61b-905e22653608-metrics-server-audit-profiles\") pod \"metrics-server-796bf69784-66sjq\" (UID: \"18595f28-dc19-4df0-a61b-905e22653608\") " pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:17.433025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.431978 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78lnx\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-kube-api-access-78lnx\") pod \"alertmanager-main-0\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:17.433332 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.433297 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ld4\" (UniqueName: \"kubernetes.io/projected/7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55-kube-api-access-44ld4\") pod \"thanos-querier-675964fd8b-v45n9\" (UID: \"7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55\") " pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:17.434241 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.434222 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7sqc\" (UniqueName: \"kubernetes.io/projected/4d408013-e118-42d1-809f-5d3b50f12565-kube-api-access-j7sqc\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.435015 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.434997 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5csc\" (UniqueName: \"kubernetes.io/projected/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-kube-api-access-h5csc\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:23:17.436615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.436579 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsfjt\" (UniqueName: \"kubernetes.io/projected/6962a1b2-43de-4da8-a1f4-af8101d42fd0-kube-api-access-lsfjt\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:23:17.437163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.437134 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2r9f\" (UniqueName: \"kubernetes.io/projected/e9ed2868-24ad-473c-bac4-01344c0e4bed-kube-api-access-c2r9f\") pod \"kube-state-metrics-5dc8d7d89b-zk4kv\" (UID: \"e9ed2868-24ad-473c-bac4-01344c0e4bed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-zk4kv" Nov 01 22:23:17.437308 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.437205 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5t7\" (UniqueName: \"kubernetes.io/projected/4dbe2700-733d-4ebd-9f47-7b9f30bb62fd-kube-api-access-7w5t7\") pod \"network-check-target-vzk9n\" (UID: \"4dbe2700-733d-4ebd-9f47-7b9f30bb62fd\") " pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:23:17.437308 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.437294 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctgxg\" (UniqueName: \"kubernetes.io/projected/26366d8c-76f1-479e-9715-6512f20bd3c6-kube-api-access-ctgxg\") pod \"dns-default-sh7c7\" (UID: \"26366d8c-76f1-479e-9715-6512f20bd3c6\") " pod="openshift-dns/dns-default-sh7c7" Nov 01 22:23:17.437587 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.437563 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgm9\" (UniqueName: \"kubernetes.io/projected/c034008e-a79f-4ebb-9316-d5d63bc1dbb9-kube-api-access-4sgm9\") pod \"telemeter-client-bdf8c4fd-s4wbr\" (UID: \"c034008e-a79f-4ebb-9316-d5d63bc1dbb9\") " pod="openshift-monitoring/telemeter-client-bdf8c4fd-s4wbr" Nov 01 22:23:17.437763 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.437745 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hsxj\" (UniqueName: \"kubernetes.io/projected/76a5a648-0585-408d-8556-c0e31dcea942-kube-api-access-2hsxj\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:23:17.438487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.438461 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5dtj\" (UniqueName: \"kubernetes.io/projected/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-kube-api-access-k5dtj\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:23:17.438686 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.438668 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c577v\" (UniqueName: \"kubernetes.io/projected/e8572e79-8cf8-4b15-b77f-ba409bd3b51f-kube-api-access-c577v\") pod \"openshift-state-metrics-7bd7489957-z8mkz\" (UID: \"e8572e79-8cf8-4b15-b77f-ba409bd3b51f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-z8mkz" Nov 01 22:23:17.438748 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.438728 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fhtq\" (UniqueName: \"kubernetes.io/projected/73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d-kube-api-access-6fhtq\") pod \"ingress-canary-77bdk\" (UID: \"73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d\") " pod="openshift-ingress-canary/ingress-canary-77bdk" Nov 01 22:23:17.532264 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532235 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-system-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.532264 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532265 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-hostroot\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532287 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-netns\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532312 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-crio-socket\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532334 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-cnibin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532365 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-host-slash\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532372 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-system-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532373 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-netns\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532388 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-root\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532401 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0-crio-socket\") pod \"insights-runtime-extractor-kc74w\" (UID: \"ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0\") " pod="openshift-insights/insights-runtime-extractor-kc74w" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532411 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-multus-certs\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532413 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1dd1e50-ecd0-4724-862d-752e9e1aecd0-host-slash\") pod \"iptables-alerter-v2bl8\" (UID: \"e1dd1e50-ecd0-4724-862d-752e9e1aecd0\") " pod="openshift-network-operator/iptables-alerter-v2bl8" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532421 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-cnibin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532428 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-node-log\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532438 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-root\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532373 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-hostroot\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532463 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-node-log\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532479 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-multus-certs\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.532500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532481 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-wtmp\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532514 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-socket-dir-parent\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532541 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532561 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-netd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532574 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-node-exporter-wtmp\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532584 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-socket-dir-parent\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532587 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532592 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-multus\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532615 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-netd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532622 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-multus\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532621 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6962a1b2-43de-4da8-a1f4-af8101d42fd0-host\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532654 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-conf-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532670 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-systemd-units\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532683 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6962a1b2-43de-4da8-a1f4-af8101d42fd0-host\") pod \"node-ca-f9snb\" (UID: \"6962a1b2-43de-4da8-a1f4-af8101d42fd0\") " pod="openshift-image-registry/node-ca-f9snb" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532686 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-sys\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532703 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-conf-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532714 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d77255dd-93d5-4c30-8a4c-f4c7ea44d702-sys\") pod \"node-exporter-cw7qc\" (UID: \"d77255dd-93d5-4c30-8a4c-f4c7ea44d702\") " pod="openshift-monitoring/node-exporter-cw7qc" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532733 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-systemd-units\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532743 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-cnibin\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532752 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-cnibin\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532768 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-kubelet\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532784 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-etc-kubernetes\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532801 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-slash\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532812 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-kubelet\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532825 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532841 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-slash\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532844 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-etc-kubernetes\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532853 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532852 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532878 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532885 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-etc-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532911 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-kubelet\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532917 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-etc-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532933 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-log-socket\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532954 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-kubelet\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532959 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-var-lib-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532989 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-log-socket\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.532989 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-system-cni-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533018 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-system-cni-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533020 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533059 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-netns\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533065 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-multus-cni-dir\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.533053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533060 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-var-lib-openvswitch\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533083 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-systemd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533090 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-netns\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533129 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76a5a648-0585-408d-8556-c0e31dcea942-hosts-file\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533131 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-systemd\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533150 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-bin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533169 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533188 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-k8s-cni-cncf-io\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533182 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76a5a648-0585-408d-8556-c0e31dcea942-hosts-file\") pod \"node-resolver-mfnxk\" (UID: \"76a5a648-0585-408d-8556-c0e31dcea942\") " pod="openshift-dns/node-resolver-mfnxk" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533215 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-bin\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533222 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-var-lib-cni-bin\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533245 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d408013-e118-42d1-809f-5d3b50f12565-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8rz8\" (UID: \"4d408013-e118-42d1-809f-5d3b50f12565\") " pod="openshift-multus/multus-additional-cni-plugins-t8rz8" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533249 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-ovn\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533259 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-host-cni-bin\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533246 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5baeb178-3616-4b1d-8515-d8ff71486666-host-run-k8s-cni-cncf-io\") pod \"multus-qj4tr\" (UID: \"5baeb178-3616-4b1d-8515-d8ff71486666\") " pod="openshift-multus/multus-qj4tr" Nov 01 22:23:17.534256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.533295 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c-run-ovn\") pod \"ovnkube-node-zntdq\" (UID: \"bd5b3c1d-6e53-422e-b15a-ca0b66d4dd2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:17.716787 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:17.716759 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fxlz9" Nov 01 22:23:19.057935 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:19.057903 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sh7c7" Nov 01 22:23:19.278801 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:19.278768 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:23:19.927013 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:19.926974 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:23:20.153626 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:20.153601 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" Nov 01 22:23:22.525204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:22.525180 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:23.411712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:23.411681 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:23.417802 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:23.417775 6466 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:23:24.767069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:24.767039 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-796bf69784-66sjq" Nov 01 22:23:29.707169 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:29.707077 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:23:29.707700 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:29.707421 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="alertmanager" containerID="cri-o://a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa" gracePeriod=120 Nov 01 22:23:29.707700 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:29.707435 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy-metric" containerID="cri-o://d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7" gracePeriod=120 Nov 01 22:23:29.707700 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:29.707460 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy" containerID="cri-o://47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0" gracePeriod=120 Nov 01 22:23:29.707700 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:29.707473 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="config-reloader" containerID="cri-o://ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab" gracePeriod=120 Nov 01 22:23:29.707700 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:29.707519 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="prom-label-proxy" containerID="cri-o://6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653" gracePeriod=120 Nov 01 22:23:29.707700 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:29.707443 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy-web" containerID="cri-o://bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1" gracePeriod=120 Nov 01 22:23:30.444855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:30.444823 6466 generic.go:356] "Generic (PLEG): container finished" podID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerID="6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653" exitCode=0 Nov 01 22:23:30.444855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:30.444851 6466 generic.go:356] "Generic (PLEG): container finished" podID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerID="d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7" exitCode=0 Nov 01 22:23:30.444855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:30.444858 6466 generic.go:356] "Generic (PLEG): container finished" podID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerID="47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0" exitCode=0 Nov 01 22:23:30.444855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:30.444864 6466 generic.go:356] "Generic (PLEG): container finished" podID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerID="ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab" exitCode=0 Nov 01 22:23:30.445255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:30.444869 6466 generic.go:356] "Generic (PLEG): container finished" podID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerID="a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa" exitCode=0 Nov 01 22:23:30.445255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:30.444895 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerDied","Data":"6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653"} Nov 01 22:23:30.445255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:30.444976 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerDied","Data":"d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7"} Nov 01 22:23:30.445255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:30.445016 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerDied","Data":"47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0"} Nov 01 22:23:30.445255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:30.445062 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerDied","Data":"ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab"} Nov 01 22:23:30.445255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:30.445095 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerDied","Data":"a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa"} Nov 01 22:23:30.946375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:30.946350 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.045559 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045481 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-tls-assets\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-tls-assets\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.045559 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045525 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-metrics-client-ca\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.045762 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045561 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-main-tls\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.045762 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045618 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.045762 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045651 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.045762 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045680 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-kube-api-access-78lnx\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-kube-api-access-78lnx\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.045762 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045712 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-out\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.045762 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045751 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.046069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045787 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-volume\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-volume\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.046069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045831 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-web\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.046069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045859 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-cluster-tls-config\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.046069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045900 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-web-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-web-config\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.046069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.045934 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-main-db\") pod \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\" (UID: \"f12c8a75-38c0-4e45-9ec9-9e625521ceb8\") " Nov 01 22:23:31.046069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.046003 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-metrics-client-ca" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:31.046069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.046026 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:31.046481 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.046252 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-metrics-client-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.046481 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.046273 6466 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.047754 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.047685 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-main-db" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:31.049024 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.048998 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-web" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:31.049174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.049075 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-tls-assets" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:31.049174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.049097 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-volume" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:31.049297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.049255 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-main-tls" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:31.049347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.049326 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:31.049517 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.049495 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-metric" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:31.050146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.050101 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-kube-api-access-78lnx" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "kube-api-access-78lnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:31.050360 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.050339 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-out" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:31.053342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.053280 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-cluster-tls-config" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:31.059906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.059884 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-web-config" pod "f12c8a75-38c0-4e45-9ec9-9e625521ceb8" (UID: "f12c8a75-38c0-4e45-9ec9-9e625521ceb8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:31.147421 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.147373 6466 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.147421 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.147419 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78lnx\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-kube-api-access-78lnx\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.147421 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.147430 6466 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-out\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.147421 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.147440 6466 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.147683 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.147450 6466 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.147683 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.147459 6466 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.147683 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.147468 6466 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-cluster-tls-config\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.147683 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.147477 6466 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-web-config\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.147683 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.147486 6466 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-alertmanager-main-db\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.147683 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.147494 6466 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-tls-assets\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.147683 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.147502 6466 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f12c8a75-38c0-4e45-9ec9-9e625521ceb8-secret-alertmanager-main-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:31.455653 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.455616 6466 generic.go:356] "Generic (PLEG): container finished" podID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerID="bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1" exitCode=0 Nov 01 22:23:31.455795 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.455698 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerDied","Data":"bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1"} Nov 01 22:23:31.455795 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.455706 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.455883 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.455800 6466 scope.go:122] "RemoveContainer" containerID="6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653" Nov 01 22:23:31.455883 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.455790 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f12c8a75-38c0-4e45-9ec9-9e625521ceb8","Type":"ContainerDied","Data":"46844a8e32c7b4cfa6f04a52ced4f17086e0311f0f1d2fa694589b7525f975b8"} Nov 01 22:23:31.464530 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.464514 6466 scope.go:122] "RemoveContainer" containerID="d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7" Nov 01 22:23:31.471721 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.471705 6466 scope.go:122] "RemoveContainer" containerID="47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0" Nov 01 22:23:31.478965 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.478899 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:23:31.479070 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.479004 6466 scope.go:122] "RemoveContainer" containerID="bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1" Nov 01 22:23:31.483938 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.483914 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:23:31.486411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.486395 6466 scope.go:122] "RemoveContainer" containerID="ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab" Nov 01 22:23:31.493371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.493352 6466 scope.go:122] "RemoveContainer" containerID="a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa" Nov 01 22:23:31.500490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.500464 6466 scope.go:122] "RemoveContainer" containerID="d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8" Nov 01 22:23:31.507136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.507118 6466 scope.go:122] "RemoveContainer" containerID="6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653" Nov 01 22:23:31.507404 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:31.507384 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653\": container with ID starting with 6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653 not found: ID does not exist" containerID="6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653" Nov 01 22:23:31.507453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.507415 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653"} err="failed to get container status \"6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653\": rpc error: code = NotFound desc = could not find container \"6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653\": container with ID starting with 6e2b8cfecfaceda8abc99b6fc71b18b3b1f49c827667c6ede52fb6394576e653 not found: ID does not exist" Nov 01 22:23:31.507453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.507449 6466 scope.go:122] "RemoveContainer" containerID="d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7" Nov 01 22:23:31.507754 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:31.507730 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7\": container with ID starting with d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7 not found: ID does not exist" containerID="d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7" Nov 01 22:23:31.507832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.507766 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7"} err="failed to get container status \"d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7\": rpc error: code = NotFound desc = could not find container \"d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7\": container with ID starting with d0b475639b2e80909875dafde538e98987347b022f52a39177f0987d23ac9df7 not found: ID does not exist" Nov 01 22:23:31.507832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.507793 6466 scope.go:122] "RemoveContainer" containerID="47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0" Nov 01 22:23:31.508183 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:31.508157 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0\": container with ID starting with 47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0 not found: ID does not exist" containerID="47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0" Nov 01 22:23:31.508451 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.508193 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0"} err="failed to get container status \"47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0\": rpc error: code = NotFound desc = could not find container \"47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0\": container with ID starting with 47c918eb6ef99c906d02d66394879a7ef5ac24682c93c312ca52669d045c74f0 not found: ID does not exist" Nov 01 22:23:31.508451 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.508216 6466 scope.go:122] "RemoveContainer" containerID="bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1" Nov 01 22:23:31.508588 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:31.508511 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1\": container with ID starting with bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1 not found: ID does not exist" containerID="bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1" Nov 01 22:23:31.508588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.508537 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1"} err="failed to get container status \"bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1\": rpc error: code = NotFound desc = could not find container \"bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1\": container with ID starting with bffcfa5797f64dcee33f6bc043fa5ee63991d1686f2608aba860ef0211286bd1 not found: ID does not exist" Nov 01 22:23:31.508588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.508557 6466 scope.go:122] "RemoveContainer" containerID="ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab" Nov 01 22:23:31.508825 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.508803 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:23:31.508825 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:31.508812 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab\": container with ID starting with ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab not found: ID does not exist" containerID="ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab" Nov 01 22:23:31.508923 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.508841 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab"} err="failed to get container status \"ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab\": rpc error: code = NotFound desc = could not find container \"ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab\": container with ID starting with ace10dac040244afa486b1c142522cc10523f9efd085b5d8e18a634d1bf4f7ab not found: ID does not exist" Nov 01 22:23:31.508923 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.508860 6466 scope.go:122] "RemoveContainer" containerID="a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa" Nov 01 22:23:31.509139 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509124 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="config-reloader" Nov 01 22:23:31.509230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509142 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="config-reloader" Nov 01 22:23:31.509230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509153 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="alertmanager" Nov 01 22:23:31.509230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509160 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="alertmanager" Nov 01 22:23:31.509230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509171 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="init-config-reloader" Nov 01 22:23:31.509230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509177 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="init-config-reloader" Nov 01 22:23:31.509230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509191 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy-metric" Nov 01 22:23:31.509230 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:31.509187 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa\": container with ID starting with a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa not found: ID does not exist" containerID="a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa" Nov 01 22:23:31.509230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509213 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa"} err="failed to get container status \"a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa\": rpc error: code = NotFound desc = could not find container \"a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa\": container with ID starting with a3d5acb110ab28d2786b885628c72d8c1a5c4c58d4e3eef7d4cc6e462b2875fa not found: ID does not exist" Nov 01 22:23:31.509230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509233 6466 scope.go:122] "RemoveContainer" containerID="d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509197 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy-metric" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509296 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509304 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509316 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy-web" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509322 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy-web" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509331 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="prom-label-proxy" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509338 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="prom-label-proxy" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509421 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509430 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy-web" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509442 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="prom-label-proxy" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509453 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="config-reloader" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509461 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="alertmanager" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509468 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="init-config-reloader" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509477 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" containerName="kube-rbac-proxy-metric" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:31.509487 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8\": container with ID starting with d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8 not found: ID does not exist" containerID="d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8" Nov 01 22:23:31.509567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.509505 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8"} err="failed to get container status \"d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8\": rpc error: code = NotFound desc = could not find container \"d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8\": container with ID starting with d4e8d821c2d67ec7e79c60c1d38e115b6f56113fc2c880264eeb0798b49233a8 not found: ID does not exist" Nov 01 22:23:31.512801 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.512783 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.516600 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.516577 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 22:23:31.516716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.516602 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 22:23:31.516716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.516632 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 22:23:31.516837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.516716 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 22:23:31.516837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.516742 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 22:23:31.516837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.516745 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 22:23:31.516837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.516784 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 22:23:31.516837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.516800 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 22:23:31.517063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.516869 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mn4xg\"" Nov 01 22:23:31.529061 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.529037 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 22:23:31.530149 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.530132 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:23:31.650896 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.650861 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-web-config\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.650896 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.650904 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93531b14-ea0b-44b7-91c6-e99c934595a1-config-out\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.651130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.650923 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.651130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.650997 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snmwf\" (UniqueName: \"kubernetes.io/projected/93531b14-ea0b-44b7-91c6-e99c934595a1-kube-api-access-snmwf\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.651130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.651039 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.651130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.651102 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93531b14-ea0b-44b7-91c6-e99c934595a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.651256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.651145 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93531b14-ea0b-44b7-91c6-e99c934595a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.651256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.651164 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.651256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.651182 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.651256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.651214 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93531b14-ea0b-44b7-91c6-e99c934595a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.651371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.651259 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.651371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.651295 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/93531b14-ea0b-44b7-91c6-e99c934595a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.651432 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.651369 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752208 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752134 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snmwf\" (UniqueName: \"kubernetes.io/projected/93531b14-ea0b-44b7-91c6-e99c934595a1-kube-api-access-snmwf\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752208 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752175 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752208 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752197 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93531b14-ea0b-44b7-91c6-e99c934595a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752216 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93531b14-ea0b-44b7-91c6-e99c934595a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752232 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752253 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752273 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93531b14-ea0b-44b7-91c6-e99c934595a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752288 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752310 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/93531b14-ea0b-44b7-91c6-e99c934595a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752355 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752387 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-web-config\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752425 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93531b14-ea0b-44b7-91c6-e99c934595a1-config-out\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752449 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.752929 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752643 6466 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 22:23:31.752929 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.752900 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/93531b14-ea0b-44b7-91c6-e99c934595a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.753157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.753134 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93531b14-ea0b-44b7-91c6-e99c934595a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.753495 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.753318 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93531b14-ea0b-44b7-91c6-e99c934595a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.755557 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.755454 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.755557 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.755483 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93531b14-ea0b-44b7-91c6-e99c934595a1-config-out\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.755720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.755598 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.755720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.755621 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.755720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.755635 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.755890 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.755721 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-web-config\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.755890 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.755876 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.756194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.756175 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93531b14-ea0b-44b7-91c6-e99c934595a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.757204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.757186 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/93531b14-ea0b-44b7-91c6-e99c934595a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.761192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.761172 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snmwf\" (UniqueName: \"kubernetes.io/projected/93531b14-ea0b-44b7-91c6-e99c934595a1-kube-api-access-snmwf\") pod \"alertmanager-main-0\" (UID: \"93531b14-ea0b-44b7-91c6-e99c934595a1\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:31.829994 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:31.829962 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:32.110965 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:32.110915 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:23:32.112466 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:23:32.112439 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93531b14_ea0b_44b7_91c6_e99c934595a1.slice/crio-7bb963294a42011dcbcce6ef5344ddd87eac650f8c8602a5a383abeb6ab4ac9e WatchSource:0}: Error finding container 7bb963294a42011dcbcce6ef5344ddd87eac650f8c8602a5a383abeb6ab4ac9e: Status 404 returned error can't find the container with id 7bb963294a42011dcbcce6ef5344ddd87eac650f8c8602a5a383abeb6ab4ac9e Nov 01 22:23:32.337520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:32.337485 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12c8a75-38c0-4e45-9ec9-9e625521ceb8" path="/var/lib/kubelet/pods/f12c8a75-38c0-4e45-9ec9-9e625521ceb8/volumes" Nov 01 22:23:32.463883 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:32.463849 6466 generic.go:356] "Generic (PLEG): container finished" podID="93531b14-ea0b-44b7-91c6-e99c934595a1" containerID="a7bd0b39b2c6e6c3c13f549f4f37a6a972b8e0bb9ab4ab250f4fa946622fb53e" exitCode=0 Nov 01 22:23:32.464076 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:32.463940 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93531b14-ea0b-44b7-91c6-e99c934595a1","Type":"ContainerDied","Data":"a7bd0b39b2c6e6c3c13f549f4f37a6a972b8e0bb9ab4ab250f4fa946622fb53e"} Nov 01 22:23:32.464076 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:32.464052 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93531b14-ea0b-44b7-91c6-e99c934595a1","Type":"ContainerStarted","Data":"7bb963294a42011dcbcce6ef5344ddd87eac650f8c8602a5a383abeb6ab4ac9e"} Nov 01 22:23:33.471999 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:33.471966 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93531b14-ea0b-44b7-91c6-e99c934595a1","Type":"ContainerStarted","Data":"51f247ca6820d7dbe523ac06349a28b683e80689d220d4027f8a8925675d7741"} Nov 01 22:23:33.472519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:33.472049 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93531b14-ea0b-44b7-91c6-e99c934595a1","Type":"ContainerStarted","Data":"bd320ca0cd40cd3bdbee4d94d1f9c7a0fc4862ebfc2791fac4fd4115c358caca"} Nov 01 22:23:33.472519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:33.472101 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93531b14-ea0b-44b7-91c6-e99c934595a1","Type":"ContainerStarted","Data":"86a8b699af3e203e8e7d88346f7438715bb33186e77bbe559eb0a1b1266e7884"} Nov 01 22:23:33.472519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:33.472168 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93531b14-ea0b-44b7-91c6-e99c934595a1","Type":"ContainerStarted","Data":"ce60bf33e1967768a1f5307b1e43b168b06781633b836e784287e6ccc0610bc0"} Nov 01 22:23:33.472519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:33.472210 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93531b14-ea0b-44b7-91c6-e99c934595a1","Type":"ContainerStarted","Data":"12a70411a357ed5059df58a545f287ac4bc28ba5877811e341108e7dd7751ca6"} Nov 01 22:23:33.472519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:33.472251 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93531b14-ea0b-44b7-91c6-e99c934595a1","Type":"ContainerStarted","Data":"6bf7657d66b3f6a157938a2ac39cc54dcfc90b43b742fb09372c01fbf29aeeb8"} Nov 01 22:23:33.506675 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:33.506616 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.494641328 podStartE2EDuration="2.506597077s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.955749ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:33.505157664 +0000 UTC m=+17.289596589" watchObservedRunningTime="2026-11-01 22:23:33.506597077 +0000 UTC m=+17.291035999" Nov 01 22:23:43.706152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:43.706126 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zntdq" Nov 01 22:23:49.278707 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:23:49.278669 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:23:49.929943 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:23:49.929911 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vzk9n" Nov 01 22:24:19.280169 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:24:19.280129 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:24:49.279054 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:24:49.279012 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:25:19.279296 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:25:19.279217 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:25:31.279566 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.279528 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc"] Nov 01 22:25:31.283027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.283010 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:31.294018 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.293992 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-5kx79\"" Nov 01 22:25:31.300934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.296700 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc"] Nov 01 22:25:31.300934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.296856 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 22:25:31.300934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.296943 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 22:25:31.351089 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.351052 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj"] Nov 01 22:25:31.354502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.354484 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:31.362939 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.362916 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj"] Nov 01 22:25:31.393696 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.393671 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-4n47r\"" Nov 01 22:25:31.404722 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.404695 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:31.404858 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.404763 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:31.404915 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.404878 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnh5\" (UniqueName: \"kubernetes.io/projected/4239b528-aee1-4b22-91d7-810c79d7d299-kube-api-access-dmnh5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:31.506166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.506139 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:31.506317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.506194 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:31.506317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.506278 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnh5\" (UniqueName: \"kubernetes.io/projected/4239b528-aee1-4b22-91d7-810c79d7d299-kube-api-access-dmnh5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:31.506402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.506329 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:31.506402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.506360 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:31.506402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.506389 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wvf\" (UniqueName: \"kubernetes.io/projected/e9598810-9cd0-471a-81a3-b3f6f9b523b2-kube-api-access-q5wvf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:31.506570 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.506551 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:31.506658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.506640 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:31.514411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.514390 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnh5\" (UniqueName: \"kubernetes.io/projected/4239b528-aee1-4b22-91d7-810c79d7d299-kube-api-access-dmnh5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:31.607237 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.607150 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:31.607402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.607302 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:31.607402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.607336 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wvf\" (UniqueName: \"kubernetes.io/projected/e9598810-9cd0-471a-81a3-b3f6f9b523b2-kube-api-access-q5wvf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:31.607402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.607362 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:31.607691 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.607668 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:31.607731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.607698 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:31.615771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.615751 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wvf\" (UniqueName: \"kubernetes.io/projected/e9598810-9cd0-471a-81a3-b3f6f9b523b2-kube-api-access-q5wvf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:31.670443 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.670407 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:31.690043 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.690017 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt"] Nov 01 22:25:31.693324 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.693304 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.703131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.703082 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt"] Nov 01 22:25:31.703361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.703315 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 22:25:31.703419 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.703409 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 22:25:31.703472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.703462 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 22:25:31.703648 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.703626 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c7qwl\"" Nov 01 22:25:31.704250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.704229 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 22:25:31.704338 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.704303 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 22:25:31.704400 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.704365 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 22:25:31.809954 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.809916 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.809954 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.809956 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qfqj\" (UniqueName: \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-kube-api-access-2qfqj\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.810195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.809978 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c027922-4282-41de-b78c-a1a1f5f129b2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.810195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.810159 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.810261 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.810222 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.810293 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.810264 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.810293 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.810288 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.887444 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.887416 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc"] Nov 01 22:25:31.890019 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:25:31.889992 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4239b528_aee1_4b22_91d7_810c79d7d299.slice/crio-d65a27116b14a9f2ad7495921b0a3246cace23c19f9d7ec335c2fda388ba575f WatchSource:0}: Error finding container d65a27116b14a9f2ad7495921b0a3246cace23c19f9d7ec335c2fda388ba575f: Status 404 returned error can't find the container with id d65a27116b14a9f2ad7495921b0a3246cace23c19f9d7ec335c2fda388ba575f Nov 01 22:25:31.891822 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.891806 6466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:25:31.910689 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.910664 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c027922-4282-41de-b78c-a1a1f5f129b2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.910804 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.910704 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.910804 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.910735 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.910804 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.910762 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.910804 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.910783 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.910994 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.910810 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.910994 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.910833 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qfqj\" (UniqueName: \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-kube-api-access-2qfqj\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.911642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.911612 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.913130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.913096 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.913207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.913128 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c027922-4282-41de-b78c-a1a1f5f129b2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.913207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.913155 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.913319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.913304 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.918990 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.918966 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qfqj\" (UniqueName: \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-kube-api-access-2qfqj\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.919509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.919490 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fqtxt\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:31.954925 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:31.954893 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj"] Nov 01 22:25:31.958277 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:25:31.958237 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9598810_9cd0_471a_81a3_b3f6f9b523b2.slice/crio-cec37d8ae63e61ee7471052a00ed18a36aa887f4873268c26ad4b8992cc73978 WatchSource:0}: Error finding container cec37d8ae63e61ee7471052a00ed18a36aa887f4873268c26ad4b8992cc73978: Status 404 returned error can't find the container with id cec37d8ae63e61ee7471052a00ed18a36aa887f4873268c26ad4b8992cc73978 Nov 01 22:25:32.012071 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:32.012046 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:32.283156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:32.283015 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt"] Nov 01 22:25:32.285629 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:25:32.285598 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c027922_4282_41de_b78c_a1a1f5f129b2.slice/crio-9e42a38d8e748a9b86f1ea7eb65328c7a2fa5249d381545ff1ee6cc2c9f83ac8 WatchSource:0}: Error finding container 9e42a38d8e748a9b86f1ea7eb65328c7a2fa5249d381545ff1ee6cc2c9f83ac8: Status 404 returned error can't find the container with id 9e42a38d8e748a9b86f1ea7eb65328c7a2fa5249d381545ff1ee6cc2c9f83ac8 Nov 01 22:25:32.897853 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:32.897813 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" event={"ID":"e9598810-9cd0-471a-81a3-b3f6f9b523b2","Type":"ContainerStarted","Data":"cec37d8ae63e61ee7471052a00ed18a36aa887f4873268c26ad4b8992cc73978"} Nov 01 22:25:32.899589 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:32.899556 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" event={"ID":"1c027922-4282-41de-b78c-a1a1f5f129b2","Type":"ContainerStarted","Data":"9e42a38d8e748a9b86f1ea7eb65328c7a2fa5249d381545ff1ee6cc2c9f83ac8"} Nov 01 22:25:32.901072 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:32.901044 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" event={"ID":"4239b528-aee1-4b22-91d7-810c79d7d299","Type":"ContainerStarted","Data":"d65a27116b14a9f2ad7495921b0a3246cace23c19f9d7ec335c2fda388ba575f"} Nov 01 22:25:36.369786 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:36.369707 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:25:36.370237 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:36.369787 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:25:36.919679 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:36.919640 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" event={"ID":"1c027922-4282-41de-b78c-a1a1f5f129b2","Type":"ContainerStarted","Data":"a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d"} Nov 01 22:25:36.919857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:36.919771 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:37.925687 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:37.925659 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:25:37.948635 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:37.948566 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" podStartSLOduration=2.867304028 podStartE2EDuration="6.948551716s" totalImagesPullingTime="4.081247688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:36.940776556 +0000 UTC m=+140.725215478" watchObservedRunningTime="2026-11-01 22:25:37.948551716 +0000 UTC m=+141.732990636" Nov 01 22:25:40.938225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:40.938189 6466 generic.go:356] "Generic (PLEG): container finished" podID="e9598810-9cd0-471a-81a3-b3f6f9b523b2" containerID="0bd2cfd30b6f2d53f1c716109ced0ba3e378abd85bb6b897ae1cbbad41e62a89" exitCode=0 Nov 01 22:25:40.938658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:40.938275 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" event={"ID":"e9598810-9cd0-471a-81a3-b3f6f9b523b2","Type":"ContainerDied","Data":"0bd2cfd30b6f2d53f1c716109ced0ba3e378abd85bb6b897ae1cbbad41e62a89"} Nov 01 22:25:40.939902 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:40.939880 6466 generic.go:356] "Generic (PLEG): container finished" podID="4239b528-aee1-4b22-91d7-810c79d7d299" containerID="b7c6526f4dd3f5a63e806ab91571b5b50c1480a42375919eefce9289f879fbf9" exitCode=0 Nov 01 22:25:40.940010 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:40.939916 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" event={"ID":"4239b528-aee1-4b22-91d7-810c79d7d299","Type":"ContainerDied","Data":"b7c6526f4dd3f5a63e806ab91571b5b50c1480a42375919eefce9289f879fbf9"} Nov 01 22:25:41.945392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:41.945308 6466 generic.go:356] "Generic (PLEG): container finished" podID="4239b528-aee1-4b22-91d7-810c79d7d299" containerID="f1900419f150179133a561917f3ed89bf897fe1b24ee7c0614c0fe417d72c7df" exitCode=0 Nov 01 22:25:41.945779 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:41.945399 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" event={"ID":"4239b528-aee1-4b22-91d7-810c79d7d299","Type":"ContainerDied","Data":"f1900419f150179133a561917f3ed89bf897fe1b24ee7c0614c0fe417d72c7df"} Nov 01 22:25:44.962597 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:44.962558 6466 generic.go:356] "Generic (PLEG): container finished" podID="e9598810-9cd0-471a-81a3-b3f6f9b523b2" containerID="23200e21a616497068f470aeb441f781077f99a7e0b71e957e777c1117c84d23" exitCode=0 Nov 01 22:25:44.962993 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:44.962642 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" event={"ID":"e9598810-9cd0-471a-81a3-b3f6f9b523b2","Type":"ContainerDied","Data":"23200e21a616497068f470aeb441f781077f99a7e0b71e957e777c1117c84d23"} Nov 01 22:25:49.288069 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:25:49.288027 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:25:50.993865 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:50.993826 6466 generic.go:356] "Generic (PLEG): container finished" podID="4239b528-aee1-4b22-91d7-810c79d7d299" containerID="c05e7410047dfee63d67780020cb73e5f9764ad9241cd10be3a4baf74c2c091b" exitCode=0 Nov 01 22:25:50.994342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:50.993877 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" event={"ID":"4239b528-aee1-4b22-91d7-810c79d7d299","Type":"ContainerDied","Data":"c05e7410047dfee63d67780020cb73e5f9764ad9241cd10be3a4baf74c2c091b"} Nov 01 22:25:52.133988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:52.133965 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:52.198215 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:52.198186 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4239b528-aee1-4b22-91d7-810c79d7d299-kube-api-access-dmnh5\" (UniqueName: \"kubernetes.io/projected/4239b528-aee1-4b22-91d7-810c79d7d299-kube-api-access-dmnh5\") pod \"4239b528-aee1-4b22-91d7-810c79d7d299\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " Nov 01 22:25:52.198355 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:52.198286 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-bundle\") pod \"4239b528-aee1-4b22-91d7-810c79d7d299\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " Nov 01 22:25:52.198355 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:52.198314 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-util\" (UniqueName: \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-util\") pod \"4239b528-aee1-4b22-91d7-810c79d7d299\" (UID: \"4239b528-aee1-4b22-91d7-810c79d7d299\") " Nov 01 22:25:52.198959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:52.198932 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-bundle" pod "4239b528-aee1-4b22-91d7-810c79d7d299" (UID: "4239b528-aee1-4b22-91d7-810c79d7d299"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:52.200487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:52.200455 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4239b528-aee1-4b22-91d7-810c79d7d299-kube-api-access-dmnh5" pod "4239b528-aee1-4b22-91d7-810c79d7d299" (UID: "4239b528-aee1-4b22-91d7-810c79d7d299"). InnerVolumeSpecName "kube-api-access-dmnh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:52.202834 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:52.202790 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-util" pod "4239b528-aee1-4b22-91d7-810c79d7d299" (UID: "4239b528-aee1-4b22-91d7-810c79d7d299"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:52.299439 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:52.299354 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmnh5\" (UniqueName: \"kubernetes.io/projected/4239b528-aee1-4b22-91d7-810c79d7d299-kube-api-access-dmnh5\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:52.299439 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:52.299387 6466 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-bundle\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:52.299439 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:52.299400 6466 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4239b528-aee1-4b22-91d7-810c79d7d299-util\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:53.003582 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:53.003556 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" Nov 01 22:25:53.003744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:53.003555 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qx9kc" event={"ID":"4239b528-aee1-4b22-91d7-810c79d7d299","Type":"ContainerDied","Data":"d65a27116b14a9f2ad7495921b0a3246cace23c19f9d7ec335c2fda388ba575f"} Nov 01 22:25:53.003744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:53.003686 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d65a27116b14a9f2ad7495921b0a3246cace23c19f9d7ec335c2fda388ba575f" Nov 01 22:25:53.005898 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:53.005863 6466 generic.go:356] "Generic (PLEG): container finished" podID="e9598810-9cd0-471a-81a3-b3f6f9b523b2" containerID="e3fe31bd2a17923dbbd5e3939f42b9ec257d193be3cced551996b3d8a29d5bc1" exitCode=0 Nov 01 22:25:53.006029 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:53.005904 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" event={"ID":"e9598810-9cd0-471a-81a3-b3f6f9b523b2","Type":"ContainerDied","Data":"e3fe31bd2a17923dbbd5e3939f42b9ec257d193be3cced551996b3d8a29d5bc1"} Nov 01 22:25:54.206755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:54.206723 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:54.316959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:54.316883 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-util\" (UniqueName: \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-util\") pod \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " Nov 01 22:25:54.316959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:54.316924 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9598810-9cd0-471a-81a3-b3f6f9b523b2-kube-api-access-q5wvf\" (UniqueName: \"kubernetes.io/projected/e9598810-9cd0-471a-81a3-b3f6f9b523b2-kube-api-access-q5wvf\") pod \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " Nov 01 22:25:54.317164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:54.317052 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-bundle\") pod \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\" (UID: \"e9598810-9cd0-471a-81a3-b3f6f9b523b2\") " Nov 01 22:25:54.317862 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:54.317836 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-bundle" pod "e9598810-9cd0-471a-81a3-b3f6f9b523b2" (UID: "e9598810-9cd0-471a-81a3-b3f6f9b523b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:54.319053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:54.319021 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9598810-9cd0-471a-81a3-b3f6f9b523b2-kube-api-access-q5wvf" pod "e9598810-9cd0-471a-81a3-b3f6f9b523b2" (UID: "e9598810-9cd0-471a-81a3-b3f6f9b523b2"). InnerVolumeSpecName "kube-api-access-q5wvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:54.321377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:54.321354 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-util" pod "e9598810-9cd0-471a-81a3-b3f6f9b523b2" (UID: "e9598810-9cd0-471a-81a3-b3f6f9b523b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:54.418181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:54.418143 6466 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-util\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:54.418332 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:54.418194 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wvf\" (UniqueName: \"kubernetes.io/projected/e9598810-9cd0-471a-81a3-b3f6f9b523b2-kube-api-access-q5wvf\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:54.418332 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:54.418206 6466 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9598810-9cd0-471a-81a3-b3f6f9b523b2-bundle\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:55.015372 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:55.015347 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" Nov 01 22:25:55.015372 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:55.015367 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccrhgj" event={"ID":"e9598810-9cd0-471a-81a3-b3f6f9b523b2","Type":"ContainerDied","Data":"cec37d8ae63e61ee7471052a00ed18a36aa887f4873268c26ad4b8992cc73978"} Nov 01 22:25:55.015605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:25:55.015410 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cec37d8ae63e61ee7471052a00ed18a36aa887f4873268c26ad4b8992cc73978" Nov 01 22:26:19.279269 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:26:19.279215 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:26:26.619394 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619312 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd"] Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619608 6466 cpu_manager.go:414] "containerMap: removing container" podUID="4239b528-aee1-4b22-91d7-810c79d7d299" containerName="util" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619619 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4239b528-aee1-4b22-91d7-810c79d7d299" containerName="util" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619627 6466 cpu_manager.go:414] "containerMap: removing container" podUID="4239b528-aee1-4b22-91d7-810c79d7d299" containerName="pull" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619632 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4239b528-aee1-4b22-91d7-810c79d7d299" containerName="pull" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619648 6466 cpu_manager.go:414] "containerMap: removing container" podUID="e9598810-9cd0-471a-81a3-b3f6f9b523b2" containerName="pull" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619653 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9598810-9cd0-471a-81a3-b3f6f9b523b2" containerName="pull" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619667 6466 cpu_manager.go:414] "containerMap: removing container" podUID="e9598810-9cd0-471a-81a3-b3f6f9b523b2" containerName="util" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619694 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9598810-9cd0-471a-81a3-b3f6f9b523b2" containerName="util" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619702 6466 cpu_manager.go:414] "containerMap: removing container" podUID="4239b528-aee1-4b22-91d7-810c79d7d299" containerName="extract" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619707 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4239b528-aee1-4b22-91d7-810c79d7d299" containerName="extract" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619714 6466 cpu_manager.go:414] "containerMap: removing container" podUID="e9598810-9cd0-471a-81a3-b3f6f9b523b2" containerName="extract" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619718 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9598810-9cd0-471a-81a3-b3f6f9b523b2" containerName="extract" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619765 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="4239b528-aee1-4b22-91d7-810c79d7d299" containerName="extract" Nov 01 22:26:26.620172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.619771 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9598810-9cd0-471a-81a3-b3f6f9b523b2" containerName="extract" Nov 01 22:26:26.621572 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.621555 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.648353 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.648319 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd"] Nov 01 22:26:26.697383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.697351 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 22:26:26.697768 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.697744 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 22:26:26.698004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.697800 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 22:26:26.698004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.697947 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 22:26:26.699016 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.698997 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 22:26:26.699272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.699248 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-fc42r\"" Nov 01 22:26:26.788031 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.787996 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf859056-1280-4da9-a120-1144db7ae3c7-metrics-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.788031 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.788034 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf859056-1280-4da9-a120-1144db7ae3c7-webhook-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.788298 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.788068 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qghk\" (UniqueName: \"kubernetes.io/projected/cf859056-1280-4da9-a120-1144db7ae3c7-kube-api-access-2qghk\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.788298 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.788236 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf859056-1280-4da9-a120-1144db7ae3c7-manifests\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.788387 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.788303 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf859056-1280-4da9-a120-1144db7ae3c7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.788387 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.788365 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf859056-1280-4da9-a120-1144db7ae3c7-tmp\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.888862 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.888831 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qghk\" (UniqueName: \"kubernetes.io/projected/cf859056-1280-4da9-a120-1144db7ae3c7-kube-api-access-2qghk\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.889039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.888874 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf859056-1280-4da9-a120-1144db7ae3c7-manifests\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.889039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.888899 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf859056-1280-4da9-a120-1144db7ae3c7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.889039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.888926 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf859056-1280-4da9-a120-1144db7ae3c7-tmp\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.889039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.888977 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf859056-1280-4da9-a120-1144db7ae3c7-metrics-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.889039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.888994 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf859056-1280-4da9-a120-1144db7ae3c7-webhook-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.889428 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.889399 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf859056-1280-4da9-a120-1144db7ae3c7-manifests\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.891211 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.891183 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf859056-1280-4da9-a120-1144db7ae3c7-tmp\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.891433 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.891415 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf859056-1280-4da9-a120-1144db7ae3c7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.891591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.891570 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf859056-1280-4da9-a120-1144db7ae3c7-metrics-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.891650 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.891603 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf859056-1280-4da9-a120-1144db7ae3c7-webhook-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.899646 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.899619 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qghk\" (UniqueName: \"kubernetes.io/projected/cf859056-1280-4da9-a120-1144db7ae3c7-kube-api-access-2qghk\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-mvdmd\" (UID: \"cf859056-1280-4da9-a120-1144db7ae3c7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:26.934031 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:26.934001 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:27.230294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:27.230267 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd"] Nov 01 22:26:27.232585 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:26:27.232559 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf859056_1280_4da9_a120_1144db7ae3c7.slice/crio-7a13c2d15561907bda30fc88f7cbc97967ef67274fb827cfb82ce5fb2e301794 WatchSource:0}: Error finding container 7a13c2d15561907bda30fc88f7cbc97967ef67274fb827cfb82ce5fb2e301794: Status 404 returned error can't find the container with id 7a13c2d15561907bda30fc88f7cbc97967ef67274fb827cfb82ce5fb2e301794 Nov 01 22:26:28.143526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:28.143469 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" event={"ID":"cf859056-1280-4da9-a120-1144db7ae3c7","Type":"ContainerStarted","Data":"7a13c2d15561907bda30fc88f7cbc97967ef67274fb827cfb82ce5fb2e301794"} Nov 01 22:26:31.159152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:31.159095 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" event={"ID":"cf859056-1280-4da9-a120-1144db7ae3c7","Type":"ContainerStarted","Data":"60a8669523fede6939e9c35887cad91f790bde3999e61c46d6debd54db8459f3"} Nov 01 22:26:32.164760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:32.164723 6466 generic.go:356] "Generic (PLEG): container finished" podID="cf859056-1280-4da9-a120-1144db7ae3c7" containerID="60a8669523fede6939e9c35887cad91f790bde3999e61c46d6debd54db8459f3" exitCode=0 Nov 01 22:26:32.165236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:32.164802 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" event={"ID":"cf859056-1280-4da9-a120-1144db7ae3c7","Type":"ContainerDied","Data":"60a8669523fede6939e9c35887cad91f790bde3999e61c46d6debd54db8459f3"} Nov 01 22:26:33.171827 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:33.171783 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" event={"ID":"cf859056-1280-4da9-a120-1144db7ae3c7","Type":"ContainerStarted","Data":"5eedc8e9b3fc0343c36d6b9f263c504a1aff91c509254b76b8acd164b8a16391"} Nov 01 22:26:33.172267 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:33.171888 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:33.207065 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:33.207013 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" podStartSLOduration=2.691506327 podStartE2EDuration="7.206984774s" totalImagesPullingTime="4.107904277s" totalInitContainerRuntime="407.57417ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:33.206430057 +0000 UTC m=+196.990868971" watchObservedRunningTime="2026-11-01 22:26:33.206984774 +0000 UTC m=+196.991423697" Nov 01 22:26:38.777446 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.777415 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h"] Nov 01 22:26:38.781356 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.781337 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" Nov 01 22:26:38.786085 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.786066 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 01 22:26:38.786998 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.786977 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1f4e4413-8e24-4a65-937d-cc4a9bc0bc25-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fqs5h\" (UID: \"1f4e4413-8e24-4a65-937d-cc4a9bc0bc25\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" Nov 01 22:26:38.787082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.787020 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x95j7\" (UniqueName: \"kubernetes.io/projected/1f4e4413-8e24-4a65-937d-cc4a9bc0bc25-kube-api-access-x95j7\") pod \"kuadrant-console-plugin-66c4fb575d-fqs5h\" (UID: \"1f4e4413-8e24-4a65-937d-cc4a9bc0bc25\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" Nov 01 22:26:38.787172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.787149 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f4e4413-8e24-4a65-937d-cc4a9bc0bc25-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fqs5h\" (UID: \"1f4e4413-8e24-4a65-937d-cc4a9bc0bc25\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" Nov 01 22:26:38.789349 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.789234 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5wxtv\"" Nov 01 22:26:38.790625 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.790607 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 22:26:38.790734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.790670 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 22:26:38.790734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.790675 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 01 22:26:38.791691 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.791671 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h"] Nov 01 22:26:38.887610 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.887578 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1f4e4413-8e24-4a65-937d-cc4a9bc0bc25-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fqs5h\" (UID: \"1f4e4413-8e24-4a65-937d-cc4a9bc0bc25\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" Nov 01 22:26:38.887776 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.887625 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x95j7\" (UniqueName: \"kubernetes.io/projected/1f4e4413-8e24-4a65-937d-cc4a9bc0bc25-kube-api-access-x95j7\") pod \"kuadrant-console-plugin-66c4fb575d-fqs5h\" (UID: \"1f4e4413-8e24-4a65-937d-cc4a9bc0bc25\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" Nov 01 22:26:38.887776 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.887662 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f4e4413-8e24-4a65-937d-cc4a9bc0bc25-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fqs5h\" (UID: \"1f4e4413-8e24-4a65-937d-cc4a9bc0bc25\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" Nov 01 22:26:38.888210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.888189 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1f4e4413-8e24-4a65-937d-cc4a9bc0bc25-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fqs5h\" (UID: \"1f4e4413-8e24-4a65-937d-cc4a9bc0bc25\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" Nov 01 22:26:38.891587 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.890752 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f4e4413-8e24-4a65-937d-cc4a9bc0bc25-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fqs5h\" (UID: \"1f4e4413-8e24-4a65-937d-cc4a9bc0bc25\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" Nov 01 22:26:38.896589 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:38.896560 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x95j7\" (UniqueName: \"kubernetes.io/projected/1f4e4413-8e24-4a65-937d-cc4a9bc0bc25-kube-api-access-x95j7\") pod \"kuadrant-console-plugin-66c4fb575d-fqs5h\" (UID: \"1f4e4413-8e24-4a65-937d-cc4a9bc0bc25\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" Nov 01 22:26:39.107662 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:39.107580 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" Nov 01 22:26:39.374577 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:39.374498 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h"] Nov 01 22:26:39.377500 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:26:39.377473 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f4e4413_8e24_4a65_937d_cc4a9bc0bc25.slice/crio-7dcf0b40739bbf9e01fdb12ef26041a979aed0e256f70377961ed7cc717b9cf5 WatchSource:0}: Error finding container 7dcf0b40739bbf9e01fdb12ef26041a979aed0e256f70377961ed7cc717b9cf5: Status 404 returned error can't find the container with id 7dcf0b40739bbf9e01fdb12ef26041a979aed0e256f70377961ed7cc717b9cf5 Nov 01 22:26:40.200701 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:40.200666 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" event={"ID":"1f4e4413-8e24-4a65-937d-cc4a9bc0bc25","Type":"ContainerStarted","Data":"7dcf0b40739bbf9e01fdb12ef26041a979aed0e256f70377961ed7cc717b9cf5"} Nov 01 22:26:44.178474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:44.178447 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-mvdmd" Nov 01 22:26:47.239623 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:47.239588 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" event={"ID":"1f4e4413-8e24-4a65-937d-cc4a9bc0bc25","Type":"ContainerStarted","Data":"08903b695d4ff6b345f2633198569d18383c3f307b8a02df3cf7086f69b75c70"} Nov 01 22:26:47.256624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:47.256580 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fqs5h" podStartSLOduration=2.464878533 podStartE2EDuration="9.256568223s" totalImagesPullingTime="6.79168969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:47.255350203 +0000 UTC m=+211.039789124" watchObservedRunningTime="2026-11-01 22:26:47.256568223 +0000 UTC m=+211.041007144" Nov 01 22:26:49.280143 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:26:49.280092 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:26:52.217247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.217212 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9"] Nov 01 22:26:52.220215 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.220197 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" Nov 01 22:26:52.225942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.225761 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 22:26:52.227225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.227205 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 22:26:52.227225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.227212 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gj492\"" Nov 01 22:26:52.231233 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.231211 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9"] Nov 01 22:26:52.303811 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.303777 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cea7bcd-f62d-4528-b266-4d3a022c7de3-tmp\") pod \"kube-auth-proxy-79d9489c85-8f7t9\" (UID: \"3cea7bcd-f62d-4528-b266-4d3a022c7de3\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" Nov 01 22:26:52.303971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.303833 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cea7bcd-f62d-4528-b266-4d3a022c7de3-tls-certs\") pod \"kube-auth-proxy-79d9489c85-8f7t9\" (UID: \"3cea7bcd-f62d-4528-b266-4d3a022c7de3\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" Nov 01 22:26:52.303971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.303960 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4phwm\" (UniqueName: \"kubernetes.io/projected/3cea7bcd-f62d-4528-b266-4d3a022c7de3-kube-api-access-4phwm\") pod \"kube-auth-proxy-79d9489c85-8f7t9\" (UID: \"3cea7bcd-f62d-4528-b266-4d3a022c7de3\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" Nov 01 22:26:52.404615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.404575 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4phwm\" (UniqueName: \"kubernetes.io/projected/3cea7bcd-f62d-4528-b266-4d3a022c7de3-kube-api-access-4phwm\") pod \"kube-auth-proxy-79d9489c85-8f7t9\" (UID: \"3cea7bcd-f62d-4528-b266-4d3a022c7de3\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" Nov 01 22:26:52.404833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.404708 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cea7bcd-f62d-4528-b266-4d3a022c7de3-tmp\") pod \"kube-auth-proxy-79d9489c85-8f7t9\" (UID: \"3cea7bcd-f62d-4528-b266-4d3a022c7de3\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" Nov 01 22:26:52.404833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.404771 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cea7bcd-f62d-4528-b266-4d3a022c7de3-tls-certs\") pod \"kube-auth-proxy-79d9489c85-8f7t9\" (UID: \"3cea7bcd-f62d-4528-b266-4d3a022c7de3\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" Nov 01 22:26:52.407675 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.407643 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cea7bcd-f62d-4528-b266-4d3a022c7de3-tmp\") pod \"kube-auth-proxy-79d9489c85-8f7t9\" (UID: \"3cea7bcd-f62d-4528-b266-4d3a022c7de3\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" Nov 01 22:26:52.407786 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.407737 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cea7bcd-f62d-4528-b266-4d3a022c7de3-tls-certs\") pod \"kube-auth-proxy-79d9489c85-8f7t9\" (UID: \"3cea7bcd-f62d-4528-b266-4d3a022c7de3\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" Nov 01 22:26:52.423503 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.423466 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4phwm\" (UniqueName: \"kubernetes.io/projected/3cea7bcd-f62d-4528-b266-4d3a022c7de3-kube-api-access-4phwm\") pod \"kube-auth-proxy-79d9489c85-8f7t9\" (UID: \"3cea7bcd-f62d-4528-b266-4d3a022c7de3\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" Nov 01 22:26:52.531157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.531075 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" Nov 01 22:26:52.816672 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.816585 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9"] Nov 01 22:26:52.820091 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:26:52.820061 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cea7bcd_f62d_4528_b266_4d3a022c7de3.slice/crio-b06736724e3be26eeef1f8c8e5d31a624343d5eb27abe85e8d87a9d8c7744472 WatchSource:0}: Error finding container b06736724e3be26eeef1f8c8e5d31a624343d5eb27abe85e8d87a9d8c7744472: Status 404 returned error can't find the container with id b06736724e3be26eeef1f8c8e5d31a624343d5eb27abe85e8d87a9d8c7744472 Nov 01 22:26:52.918915 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.918887 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dzxct"] Nov 01 22:26:52.922379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.922360 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:26:52.934706 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.934135 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 22:26:52.935196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:52.935175 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dzxct"] Nov 01 22:26:53.010272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:53.010238 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/19de28b9-6aea-4e0d-82d9-558cff53d42f-config-file\") pod \"limitador-limitador-74dfcfb894-dzxct\" (UID: \"19de28b9-6aea-4e0d-82d9-558cff53d42f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:26:53.010433 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:53.010293 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkq6t\" (UniqueName: \"kubernetes.io/projected/19de28b9-6aea-4e0d-82d9-558cff53d42f-kube-api-access-vkq6t\") pod \"limitador-limitador-74dfcfb894-dzxct\" (UID: \"19de28b9-6aea-4e0d-82d9-558cff53d42f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:26:53.024421 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:53.024393 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dzxct"] Nov 01 22:26:53.111055 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:53.110973 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/19de28b9-6aea-4e0d-82d9-558cff53d42f-config-file\") pod \"limitador-limitador-74dfcfb894-dzxct\" (UID: \"19de28b9-6aea-4e0d-82d9-558cff53d42f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:26:53.111055 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:53.111017 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkq6t\" (UniqueName: \"kubernetes.io/projected/19de28b9-6aea-4e0d-82d9-558cff53d42f-kube-api-access-vkq6t\") pod \"limitador-limitador-74dfcfb894-dzxct\" (UID: \"19de28b9-6aea-4e0d-82d9-558cff53d42f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:26:53.111660 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:53.111629 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/19de28b9-6aea-4e0d-82d9-558cff53d42f-config-file\") pod \"limitador-limitador-74dfcfb894-dzxct\" (UID: \"19de28b9-6aea-4e0d-82d9-558cff53d42f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:26:53.125424 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:53.125395 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkq6t\" (UniqueName: \"kubernetes.io/projected/19de28b9-6aea-4e0d-82d9-558cff53d42f-kube-api-access-vkq6t\") pod \"limitador-limitador-74dfcfb894-dzxct\" (UID: \"19de28b9-6aea-4e0d-82d9-558cff53d42f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:26:53.240890 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:53.240854 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:26:53.271578 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:53.271537 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" event={"ID":"3cea7bcd-f62d-4528-b266-4d3a022c7de3","Type":"ContainerStarted","Data":"b06736724e3be26eeef1f8c8e5d31a624343d5eb27abe85e8d87a9d8c7744472"} Nov 01 22:26:53.510236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:53.510204 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dzxct"] Nov 01 22:26:53.513667 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:26:53.513636 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19de28b9_6aea_4e0d_82d9_558cff53d42f.slice/crio-653335ae853c422d444a405d40a8de5ef25bb6abccf711639f152f2660cea9fc WatchSource:0}: Error finding container 653335ae853c422d444a405d40a8de5ef25bb6abccf711639f152f2660cea9fc: Status 404 returned error can't find the container with id 653335ae853c422d444a405d40a8de5ef25bb6abccf711639f152f2660cea9fc Nov 01 22:26:54.085236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:54.085199 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zxb9h"] Nov 01 22:26:54.088426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:54.088399 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" Nov 01 22:26:54.097506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:54.097475 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zxb9h"] Nov 01 22:26:54.098253 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:54.098231 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-brn77\"" Nov 01 22:26:54.121482 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:54.121451 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqht2\" (UniqueName: \"kubernetes.io/projected/4c693bf5-4a52-43c1-b33a-9450328312e9-kube-api-access-tqht2\") pod \"authorino-56c6c9ddd5-zxb9h\" (UID: \"4c693bf5-4a52-43c1-b33a-9450328312e9\") " pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" Nov 01 22:26:54.222418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:54.222377 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqht2\" (UniqueName: \"kubernetes.io/projected/4c693bf5-4a52-43c1-b33a-9450328312e9-kube-api-access-tqht2\") pod \"authorino-56c6c9ddd5-zxb9h\" (UID: \"4c693bf5-4a52-43c1-b33a-9450328312e9\") " pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" Nov 01 22:26:54.231523 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:54.231490 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqht2\" (UniqueName: \"kubernetes.io/projected/4c693bf5-4a52-43c1-b33a-9450328312e9-kube-api-access-tqht2\") pod \"authorino-56c6c9ddd5-zxb9h\" (UID: \"4c693bf5-4a52-43c1-b33a-9450328312e9\") " pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" Nov 01 22:26:54.280008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:54.279971 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" event={"ID":"19de28b9-6aea-4e0d-82d9-558cff53d42f","Type":"ContainerStarted","Data":"653335ae853c422d444a405d40a8de5ef25bb6abccf711639f152f2660cea9fc"} Nov 01 22:26:54.402895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:54.402858 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" Nov 01 22:26:54.686312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:54.686238 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zxb9h"] Nov 01 22:26:54.693917 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:26:54.693877 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c693bf5_4a52_43c1_b33a_9450328312e9.slice/crio-eb8fbbb5e355630d138eda7dda713342ba7deea13c31c2e74ce816c2cd5dc351 WatchSource:0}: Error finding container eb8fbbb5e355630d138eda7dda713342ba7deea13c31c2e74ce816c2cd5dc351: Status 404 returned error can't find the container with id eb8fbbb5e355630d138eda7dda713342ba7deea13c31c2e74ce816c2cd5dc351 Nov 01 22:26:55.286295 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:55.286254 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" event={"ID":"4c693bf5-4a52-43c1-b33a-9450328312e9","Type":"ContainerStarted","Data":"eb8fbbb5e355630d138eda7dda713342ba7deea13c31c2e74ce816c2cd5dc351"} Nov 01 22:26:56.004857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.004822 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49"] Nov 01 22:26:56.008208 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.008162 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.012750 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.012722 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-9cjh5\"" Nov 01 22:26:56.017160 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.017137 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49"] Nov 01 22:26:56.022434 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.022146 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 22:26:56.022434 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.022281 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 22:26:56.027480 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.027453 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 01 22:26:56.041502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.039830 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56efc165-48d0-46f6-ad1a-f6cf91fc3833-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.041502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.039885 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhls\" (UniqueName: \"kubernetes.io/projected/56efc165-48d0-46f6-ad1a-f6cf91fc3833-kube-api-access-rhhls\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.041502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.039977 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56efc165-48d0-46f6-ad1a-f6cf91fc3833-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.041502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.040026 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56efc165-48d0-46f6-ad1a-f6cf91fc3833-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.141120 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.141071 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhls\" (UniqueName: \"kubernetes.io/projected/56efc165-48d0-46f6-ad1a-f6cf91fc3833-kube-api-access-rhhls\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.141120 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.141120 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56efc165-48d0-46f6-ad1a-f6cf91fc3833-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.141344 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.141147 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56efc165-48d0-46f6-ad1a-f6cf91fc3833-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.141344 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.141200 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56efc165-48d0-46f6-ad1a-f6cf91fc3833-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.141609 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.141591 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56efc165-48d0-46f6-ad1a-f6cf91fc3833-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.141732 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.141711 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56efc165-48d0-46f6-ad1a-f6cf91fc3833-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.142123 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.142086 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56efc165-48d0-46f6-ad1a-f6cf91fc3833-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.151132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.150525 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhls\" (UniqueName: \"kubernetes.io/projected/56efc165-48d0-46f6-ad1a-f6cf91fc3833-kube-api-access-rhhls\") pod \"kserve-module-controller-manager-7566cf5fc8-x5z49\" (UID: \"56efc165-48d0-46f6-ad1a-f6cf91fc3833\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.294373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.294296 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" event={"ID":"3cea7bcd-f62d-4528-b266-4d3a022c7de3","Type":"ContainerStarted","Data":"a3750f242c0735e3050eb26c824a8b4fb86f7b4ed909c47047e66b438fb4cd5e"} Nov 01 22:26:56.324315 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.324249 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79d9489c85-8f7t9" podStartSLOduration=1.150359145 podStartE2EDuration="4.324227012s" totalImagesPullingTime="3.173867867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:56.322828554 +0000 UTC m=+220.107267476" watchObservedRunningTime="2026-11-01 22:26:56.324227012 +0000 UTC m=+220.108665933" Nov 01 22:26:56.344883 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.344851 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:26:56.663273 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:56.663243 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49"] Nov 01 22:26:56.666074 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:26:56.666043 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56efc165_48d0_46f6_ad1a_f6cf91fc3833.slice/crio-3a72abc62b518c013988e8a2fd6b61ff6957ae3722c2fcb29b304a555e51c8e5 WatchSource:0}: Error finding container 3a72abc62b518c013988e8a2fd6b61ff6957ae3722c2fcb29b304a555e51c8e5: Status 404 returned error can't find the container with id 3a72abc62b518c013988e8a2fd6b61ff6957ae3722c2fcb29b304a555e51c8e5 Nov 01 22:26:57.301201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:57.301094 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" event={"ID":"56efc165-48d0-46f6-ad1a-f6cf91fc3833","Type":"ContainerStarted","Data":"3a72abc62b518c013988e8a2fd6b61ff6957ae3722c2fcb29b304a555e51c8e5"} Nov 01 22:26:57.303182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:57.303152 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" event={"ID":"19de28b9-6aea-4e0d-82d9-558cff53d42f","Type":"ContainerStarted","Data":"04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de"} Nov 01 22:26:57.319958 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:57.319865 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" podStartSLOduration=1.900768453 podStartE2EDuration="5.319825168s" totalImagesPullingTime="3.419056715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:57.318960104 +0000 UTC m=+221.103399026" watchObservedRunningTime="2026-11-01 22:26:57.319825168 +0000 UTC m=+221.104264089" Nov 01 22:26:58.311061 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:58.311022 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" event={"ID":"4c693bf5-4a52-43c1-b33a-9450328312e9","Type":"ContainerStarted","Data":"6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562"} Nov 01 22:26:58.311571 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:58.311490 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:26:58.326703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:58.326648 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" podStartSLOduration=0.832861324 podStartE2EDuration="4.326633586s" totalImagesPullingTime="3.493772262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:58.325702392 +0000 UTC m=+222.110141313" watchObservedRunningTime="2026-11-01 22:26:58.326633586 +0000 UTC m=+222.111072508" Nov 01 22:26:59.327005 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:59.326903 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" event={"ID":"56efc165-48d0-46f6-ad1a-f6cf91fc3833","Type":"ContainerStarted","Data":"35cb1de6b45bf3653b1448b6e113f52e41d74d1110932ba95bf836881e859c79"} Nov 01 22:26:59.327513 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:26:59.327140 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:27:06.767437 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:06.767364 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" podStartSLOduration=9.549244782 podStartE2EDuration="11.767352057s" totalImagesPullingTime="2.218107275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:59.355493563 +0000 UTC m=+223.139932487" watchObservedRunningTime="2026-11-01 22:27:06.767352057 +0000 UTC m=+230.551790991" Nov 01 22:27:06.768822 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:06.768173 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dzxct"] Nov 01 22:27:06.768822 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:06.768376 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" podUID="19de28b9-6aea-4e0d-82d9-558cff53d42f" containerName="limitador" containerID="cri-o://04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de" gracePeriod=30 Nov 01 22:27:06.769409 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:06.769174 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:27:07.319320 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.319293 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:27:07.365670 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.365580 6466 generic.go:356] "Generic (PLEG): container finished" podID="19de28b9-6aea-4e0d-82d9-558cff53d42f" containerID="04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de" exitCode=0 Nov 01 22:27:07.365670 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.365628 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" Nov 01 22:27:07.365670 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.365653 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" event={"ID":"19de28b9-6aea-4e0d-82d9-558cff53d42f","Type":"ContainerDied","Data":"04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de"} Nov 01 22:27:07.365943 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.365707 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-dzxct" event={"ID":"19de28b9-6aea-4e0d-82d9-558cff53d42f","Type":"ContainerDied","Data":"653335ae853c422d444a405d40a8de5ef25bb6abccf711639f152f2660cea9fc"} Nov 01 22:27:07.365943 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.365735 6466 scope.go:122] "RemoveContainer" containerID="04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de" Nov 01 22:27:07.374104 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.374086 6466 scope.go:122] "RemoveContainer" containerID="04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de" Nov 01 22:27:07.374442 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:27:07.374421 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de\": container with ID starting with 04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de not found: ID does not exist" containerID="04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de" Nov 01 22:27:07.374495 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.374453 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de"} err="failed to get container status \"04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de\": rpc error: code = NotFound desc = could not find container \"04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de\": container with ID starting with 04620ad08b02ee0c26714a013d61ae8d146e776fc81df80f94f596c35f4c19de not found: ID does not exist" Nov 01 22:27:07.440980 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.440941 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19de28b9-6aea-4e0d-82d9-558cff53d42f-config-file\" (UniqueName: \"kubernetes.io/configmap/19de28b9-6aea-4e0d-82d9-558cff53d42f-config-file\") pod \"19de28b9-6aea-4e0d-82d9-558cff53d42f\" (UID: \"19de28b9-6aea-4e0d-82d9-558cff53d42f\") " Nov 01 22:27:07.441231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.441037 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19de28b9-6aea-4e0d-82d9-558cff53d42f-kube-api-access-vkq6t\" (UniqueName: \"kubernetes.io/projected/19de28b9-6aea-4e0d-82d9-558cff53d42f-kube-api-access-vkq6t\") pod \"19de28b9-6aea-4e0d-82d9-558cff53d42f\" (UID: \"19de28b9-6aea-4e0d-82d9-558cff53d42f\") " Nov 01 22:27:07.441438 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.441413 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19de28b9-6aea-4e0d-82d9-558cff53d42f-config-file" pod "19de28b9-6aea-4e0d-82d9-558cff53d42f" (UID: "19de28b9-6aea-4e0d-82d9-558cff53d42f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:07.443225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.443198 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19de28b9-6aea-4e0d-82d9-558cff53d42f-kube-api-access-vkq6t" pod "19de28b9-6aea-4e0d-82d9-558cff53d42f" (UID: "19de28b9-6aea-4e0d-82d9-558cff53d42f"). InnerVolumeSpecName "kube-api-access-vkq6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:07.541904 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.541864 6466 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/19de28b9-6aea-4e0d-82d9-558cff53d42f-config-file\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:07.541904 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.541894 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkq6t\" (UniqueName: \"kubernetes.io/projected/19de28b9-6aea-4e0d-82d9-558cff53d42f-kube-api-access-vkq6t\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:07.687007 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.686976 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dzxct"] Nov 01 22:27:07.690473 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:07.690449 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-dzxct"] Nov 01 22:27:08.340656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:08.340611 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19de28b9-6aea-4e0d-82d9-558cff53d42f" path="/var/lib/kubelet/pods/19de28b9-6aea-4e0d-82d9-558cff53d42f/volumes" Nov 01 22:27:19.280764 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:27:19.280723 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:27:20.332569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:20.332531 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-x5z49" Nov 01 22:27:49.280421 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:27:49.280383 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:27:49.754968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.754936 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ngj6m"] Nov 01 22:27:49.755309 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.755296 6466 cpu_manager.go:414] "containerMap: removing container" podUID="19de28b9-6aea-4e0d-82d9-558cff53d42f" containerName="limitador" Nov 01 22:27:49.755362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.755311 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19de28b9-6aea-4e0d-82d9-558cff53d42f" containerName="limitador" Nov 01 22:27:49.755397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.755374 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="19de28b9-6aea-4e0d-82d9-558cff53d42f" containerName="limitador" Nov 01 22:27:49.764079 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.764055 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" Nov 01 22:27:49.771157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.771102 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ngj6m"] Nov 01 22:27:49.771337 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.771317 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 01 22:27:49.771599 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.771577 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7kqdw\"" Nov 01 22:27:49.806320 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.806291 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51eead1c-88d4-4e16-b360-6f370b3041b8-cert\") pod \"kserve-controller-manager-776cc66d47-ngj6m\" (UID: \"51eead1c-88d4-4e16-b360-6f370b3041b8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" Nov 01 22:27:49.806458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.806342 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgkjn\" (UniqueName: \"kubernetes.io/projected/51eead1c-88d4-4e16-b360-6f370b3041b8-kube-api-access-fgkjn\") pod \"kserve-controller-manager-776cc66d47-ngj6m\" (UID: \"51eead1c-88d4-4e16-b360-6f370b3041b8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" Nov 01 22:27:49.843728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.843699 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dcdfx"] Nov 01 22:27:49.846014 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.845992 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" Nov 01 22:27:49.871666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.871635 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 01 22:27:49.875472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.875437 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-cczch\"" Nov 01 22:27:49.876036 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.876005 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dcdfx"] Nov 01 22:27:49.906910 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.906884 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51eead1c-88d4-4e16-b360-6f370b3041b8-cert\") pod \"kserve-controller-manager-776cc66d47-ngj6m\" (UID: \"51eead1c-88d4-4e16-b360-6f370b3041b8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" Nov 01 22:27:49.907142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.906924 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc-tls-certs\") pod \"model-serving-api-75bd89fdfc-dcdfx\" (UID: \"cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" Nov 01 22:27:49.907142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.906984 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgkjn\" (UniqueName: \"kubernetes.io/projected/51eead1c-88d4-4e16-b360-6f370b3041b8-kube-api-access-fgkjn\") pod \"kserve-controller-manager-776cc66d47-ngj6m\" (UID: \"51eead1c-88d4-4e16-b360-6f370b3041b8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" Nov 01 22:27:49.907142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.907079 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59c8t\" (UniqueName: \"kubernetes.io/projected/cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc-kube-api-access-59c8t\") pod \"model-serving-api-75bd89fdfc-dcdfx\" (UID: \"cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" Nov 01 22:27:49.908064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.908040 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-rs5t6"] Nov 01 22:27:49.910365 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.910341 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51eead1c-88d4-4e16-b360-6f370b3041b8-cert\") pod \"kserve-controller-manager-776cc66d47-ngj6m\" (UID: \"51eead1c-88d4-4e16-b360-6f370b3041b8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" Nov 01 22:27:49.911212 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.911184 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:49.938833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.938805 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgkjn\" (UniqueName: \"kubernetes.io/projected/51eead1c-88d4-4e16-b360-6f370b3041b8-kube-api-access-fgkjn\") pod \"kserve-controller-manager-776cc66d47-ngj6m\" (UID: \"51eead1c-88d4-4e16-b360-6f370b3041b8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" Nov 01 22:27:49.955027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.954998 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-rs5t6"] Nov 01 22:27:49.973749 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.973719 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 01 22:27:49.973904 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.973818 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 01 22:27:49.973904 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:49.973833 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dnjfm\"" Nov 01 22:27:50.008533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.008444 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59c8t\" (UniqueName: \"kubernetes.io/projected/cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc-kube-api-access-59c8t\") pod \"model-serving-api-75bd89fdfc-dcdfx\" (UID: \"cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" Nov 01 22:27:50.008533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.008488 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdqvz\" (UniqueName: \"kubernetes.io/projected/b2094375-d4ac-4cf1-a8da-bb4e90ba110c-kube-api-access-pdqvz\") pod \"odh-model-controller-769875db56-rs5t6\" (UID: \"b2094375-d4ac-4cf1-a8da-bb4e90ba110c\") " pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:50.008760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.008538 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2094375-d4ac-4cf1-a8da-bb4e90ba110c-metrics-certs\") pod \"odh-model-controller-769875db56-rs5t6\" (UID: \"b2094375-d4ac-4cf1-a8da-bb4e90ba110c\") " pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:50.008760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.008563 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc-tls-certs\") pod \"model-serving-api-75bd89fdfc-dcdfx\" (UID: \"cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" Nov 01 22:27:50.008760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.008585 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2094375-d4ac-4cf1-a8da-bb4e90ba110c-cert\") pod \"odh-model-controller-769875db56-rs5t6\" (UID: \"b2094375-d4ac-4cf1-a8da-bb4e90ba110c\") " pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:50.011412 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.011381 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc-tls-certs\") pod \"model-serving-api-75bd89fdfc-dcdfx\" (UID: \"cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" Nov 01 22:27:50.023280 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.023253 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59c8t\" (UniqueName: \"kubernetes.io/projected/cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc-kube-api-access-59c8t\") pod \"model-serving-api-75bd89fdfc-dcdfx\" (UID: \"cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" Nov 01 22:27:50.078604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.078578 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" Nov 01 22:27:50.110287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.110251 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdqvz\" (UniqueName: \"kubernetes.io/projected/b2094375-d4ac-4cf1-a8da-bb4e90ba110c-kube-api-access-pdqvz\") pod \"odh-model-controller-769875db56-rs5t6\" (UID: \"b2094375-d4ac-4cf1-a8da-bb4e90ba110c\") " pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:50.110455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.110335 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2094375-d4ac-4cf1-a8da-bb4e90ba110c-metrics-certs\") pod \"odh-model-controller-769875db56-rs5t6\" (UID: \"b2094375-d4ac-4cf1-a8da-bb4e90ba110c\") " pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:50.110455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.110374 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2094375-d4ac-4cf1-a8da-bb4e90ba110c-cert\") pod \"odh-model-controller-769875db56-rs5t6\" (UID: \"b2094375-d4ac-4cf1-a8da-bb4e90ba110c\") " pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:50.112975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.112949 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2094375-d4ac-4cf1-a8da-bb4e90ba110c-cert\") pod \"odh-model-controller-769875db56-rs5t6\" (UID: \"b2094375-d4ac-4cf1-a8da-bb4e90ba110c\") " pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:50.113180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.112990 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2094375-d4ac-4cf1-a8da-bb4e90ba110c-metrics-certs\") pod \"odh-model-controller-769875db56-rs5t6\" (UID: \"b2094375-d4ac-4cf1-a8da-bb4e90ba110c\") " pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:50.121834 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.121803 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdqvz\" (UniqueName: \"kubernetes.io/projected/b2094375-d4ac-4cf1-a8da-bb4e90ba110c-kube-api-access-pdqvz\") pod \"odh-model-controller-769875db56-rs5t6\" (UID: \"b2094375-d4ac-4cf1-a8da-bb4e90ba110c\") " pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:50.157760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.157732 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" Nov 01 22:27:50.229608 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.229578 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:50.282288 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.282252 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv"] Nov 01 22:27:50.287449 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.287425 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.303005 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.302954 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv"] Nov 01 22:27:50.312733 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.312701 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6312a0f9-4dea-44d8-94a5-35f1ee409f10-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.312905 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.312821 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6312a0f9-4dea-44d8-94a5-35f1ee409f10-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.312905 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.312853 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.312905 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.312891 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.313084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.312929 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvtx\" (UniqueName: \"kubernetes.io/projected/6312a0f9-4dea-44d8-94a5-35f1ee409f10-kube-api-access-zlvtx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.313084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.312966 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.313084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.313049 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.360565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.360528 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ngj6m"] Nov 01 22:27:50.414049 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.414015 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.414049 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.414053 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6312a0f9-4dea-44d8-94a5-35f1ee409f10-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.414373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.414161 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6312a0f9-4dea-44d8-94a5-35f1ee409f10-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.414373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.414190 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.414373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.414222 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.414373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.414250 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvtx\" (UniqueName: \"kubernetes.io/projected/6312a0f9-4dea-44d8-94a5-35f1ee409f10-kube-api-access-zlvtx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.414373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.414291 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.414941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.414912 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.416879 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.416853 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6312a0f9-4dea-44d8-94a5-35f1ee409f10-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.417091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.417069 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.417191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.417164 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6312a0f9-4dea-44d8-94a5-35f1ee409f10-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.418453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.418434 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.432034 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.432010 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvtx\" (UniqueName: \"kubernetes.io/projected/6312a0f9-4dea-44d8-94a5-35f1ee409f10-kube-api-access-zlvtx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.437518 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.437494 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dcdfx"] Nov 01 22:27:50.437915 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:27:50.437890 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd830101_1bd1_4ce9_aae2_eeac9ca7d5fc.slice/crio-78e9fa19a6d3522f6fe1e11510ddbaa49374889b22917361ea3e516bc788ae8d WatchSource:0}: Error finding container 78e9fa19a6d3522f6fe1e11510ddbaa49374889b22917361ea3e516bc788ae8d: Status 404 returned error can't find the container with id 78e9fa19a6d3522f6fe1e11510ddbaa49374889b22917361ea3e516bc788ae8d Nov 01 22:27:50.438627 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.438545 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6312a0f9-4dea-44d8-94a5-35f1ee409f10-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vpkhv\" (UID: \"6312a0f9-4dea-44d8-94a5-35f1ee409f10\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.504254 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.504221 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-rs5t6"] Nov 01 22:27:50.507383 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:27:50.507357 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2094375_d4ac_4cf1_a8da_bb4e90ba110c.slice/crio-528d644a95d80708e4767c7e1fc521ddb8850253e59d84e49ff9474bac1c827c WatchSource:0}: Error finding container 528d644a95d80708e4767c7e1fc521ddb8850253e59d84e49ff9474bac1c827c: Status 404 returned error can't find the container with id 528d644a95d80708e4767c7e1fc521ddb8850253e59d84e49ff9474bac1c827c Nov 01 22:27:50.560947 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.560855 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" event={"ID":"51eead1c-88d4-4e16-b360-6f370b3041b8","Type":"ContainerStarted","Data":"28a40c0f6bd7d3d3ca02832b83ad7b5ae4010f8e4abf98a7bcaaac6af5a22b09"} Nov 01 22:27:50.562503 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.562468 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-rs5t6" event={"ID":"b2094375-d4ac-4cf1-a8da-bb4e90ba110c","Type":"ContainerStarted","Data":"528d644a95d80708e4767c7e1fc521ddb8850253e59d84e49ff9474bac1c827c"} Nov 01 22:27:50.563699 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.563671 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" event={"ID":"cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc","Type":"ContainerStarted","Data":"78e9fa19a6d3522f6fe1e11510ddbaa49374889b22917361ea3e516bc788ae8d"} Nov 01 22:27:50.600825 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.600799 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:50.883093 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.883016 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv"] Nov 01 22:27:50.885831 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:27:50.885803 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6312a0f9_4dea_44d8_94a5_35f1ee409f10.slice/crio-da437625ba1e119507c6e7332670f101bbbb69b60a8315098261cf515e844ba7 WatchSource:0}: Error finding container da437625ba1e119507c6e7332670f101bbbb69b60a8315098261cf515e844ba7: Status 404 returned error can't find the container with id da437625ba1e119507c6e7332670f101bbbb69b60a8315098261cf515e844ba7 Nov 01 22:27:50.888199 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.888167 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:27:50.888275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:50.888227 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:27:51.573134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:51.572376 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" event={"ID":"6312a0f9-4dea-44d8-94a5-35f1ee409f10","Type":"ContainerStarted","Data":"088b09a324d5409ee7e74ec9e8b215e4ac5a7c848cb8d6b7f4919259e5e01e38"} Nov 01 22:27:51.573134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:51.572442 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" event={"ID":"6312a0f9-4dea-44d8-94a5-35f1ee409f10","Type":"ContainerStarted","Data":"da437625ba1e119507c6e7332670f101bbbb69b60a8315098261cf515e844ba7"} Nov 01 22:27:51.574127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:51.574040 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:51.576226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:51.576186 6466 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vpkhv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:27:51.576708 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:51.576453 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" podUID="6312a0f9-4dea-44d8-94a5-35f1ee409f10" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:27:51.602978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:51.602919 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" podStartSLOduration=1.6028996100000001 podStartE2EDuration="1.60289961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:51.600422636 +0000 UTC m=+275.384861559" watchObservedRunningTime="2026-11-01 22:27:51.60289961 +0000 UTC m=+275.387338529" Nov 01 22:27:52.583166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:52.582932 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vpkhv" Nov 01 22:27:52.634677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:52.634638 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt"] Nov 01 22:27:52.634901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:52.634877 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" podUID="1c027922-4282-41de-b78c-a1a1f5f129b2" containerName="discovery" containerID="cri-o://a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d" gracePeriod=30 Nov 01 22:27:52.942676 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:52.942389 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:27:53.051009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.050329 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-dns-cert\") pod \"1c027922-4282-41de-b78c-a1a1f5f129b2\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " Nov 01 22:27:53.051009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.050387 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-kubeconfig\") pod \"1c027922-4282-41de-b78c-a1a1f5f129b2\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " Nov 01 22:27:53.051009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.050413 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-ca-configmap\") pod \"1c027922-4282-41de-b78c-a1a1f5f129b2\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " Nov 01 22:27:53.051009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.050451 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-cacerts\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-cacerts\") pod \"1c027922-4282-41de-b78c-a1a1f5f129b2\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " Nov 01 22:27:53.051009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.050471 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c027922-4282-41de-b78c-a1a1f5f129b2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c027922-4282-41de-b78c-a1a1f5f129b2-local-certs\") pod \"1c027922-4282-41de-b78c-a1a1f5f129b2\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " Nov 01 22:27:53.051009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.050556 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-token\" (UniqueName: \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-token\") pod \"1c027922-4282-41de-b78c-a1a1f5f129b2\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " Nov 01 22:27:53.051009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.050578 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-kube-api-access-2qfqj\" (UniqueName: \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-kube-api-access-2qfqj\") pod \"1c027922-4282-41de-b78c-a1a1f5f129b2\" (UID: \"1c027922-4282-41de-b78c-a1a1f5f129b2\") " Nov 01 22:27:53.051775 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.051638 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-ca-configmap" pod "1c027922-4282-41de-b78c-a1a1f5f129b2" (UID: "1c027922-4282-41de-b78c-a1a1f5f129b2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:53.054006 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.053975 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c027922-4282-41de-b78c-a1a1f5f129b2-local-certs" pod "1c027922-4282-41de-b78c-a1a1f5f129b2" (UID: "1c027922-4282-41de-b78c-a1a1f5f129b2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:27:53.055571 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.055546 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-cacerts" pod "1c027922-4282-41de-b78c-a1a1f5f129b2" (UID: "1c027922-4282-41de-b78c-a1a1f5f129b2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:53.055675 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.055654 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-dns-cert" pod "1c027922-4282-41de-b78c-a1a1f5f129b2" (UID: "1c027922-4282-41de-b78c-a1a1f5f129b2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:53.057300 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.057271 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-kube-api-access-2qfqj" pod "1c027922-4282-41de-b78c-a1a1f5f129b2" (UID: "1c027922-4282-41de-b78c-a1a1f5f129b2"). InnerVolumeSpecName "kube-api-access-2qfqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:53.057414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.057395 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-kubeconfig" pod "1c027922-4282-41de-b78c-a1a1f5f129b2" (UID: "1c027922-4282-41de-b78c-a1a1f5f129b2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:53.059950 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.059916 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-token" pod "1c027922-4282-41de-b78c-a1a1f5f129b2" (UID: "1c027922-4282-41de-b78c-a1a1f5f129b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:53.151533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.151497 6466 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c027922-4282-41de-b78c-a1a1f5f129b2-local-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:53.151533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.151531 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:53.151794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.151544 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qfqj\" (UniqueName: \"kubernetes.io/projected/1c027922-4282-41de-b78c-a1a1f5f129b2-kube-api-access-2qfqj\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:53.151794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.151560 6466 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-dns-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:53.151794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.151576 6466 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-kubeconfig\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:53.151794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.151589 6466 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1c027922-4282-41de-b78c-a1a1f5f129b2-istio-csr-ca-configmap\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:53.151794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.151599 6466 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1c027922-4282-41de-b78c-a1a1f5f129b2-cacerts\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:53.590097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.590060 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" event={"ID":"cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc","Type":"ContainerStarted","Data":"ecaf1d086dd81d3fc4676986c57827f38ac152e0ff25a9d520d7fd2ceefc89c1"} Nov 01 22:27:53.590638 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.590193 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" Nov 01 22:27:53.591806 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.591780 6466 generic.go:356] "Generic (PLEG): container finished" podID="1c027922-4282-41de-b78c-a1a1f5f129b2" containerID="a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d" exitCode=0 Nov 01 22:27:53.591912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.591834 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" Nov 01 22:27:53.591912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.591871 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" event={"ID":"1c027922-4282-41de-b78c-a1a1f5f129b2","Type":"ContainerDied","Data":"a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d"} Nov 01 22:27:53.591912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.591902 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" event={"ID":"1c027922-4282-41de-b78c-a1a1f5f129b2","Type":"ContainerDied","Data":"9e42a38d8e748a9b86f1ea7eb65328c7a2fa5249d381545ff1ee6cc2c9f83ac8"} Nov 01 22:27:53.592010 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.591921 6466 scope.go:122] "RemoveContainer" containerID="a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d" Nov 01 22:27:53.603020 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.602999 6466 scope.go:122] "RemoveContainer" containerID="a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d" Nov 01 22:27:53.603357 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:27:53.603335 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d\": container with ID starting with a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d not found: ID does not exist" containerID="a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d" Nov 01 22:27:53.603463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.603365 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d"} err="failed to get container status \"a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d\": rpc error: code = NotFound desc = could not find container \"a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d\": container with ID starting with a24a2da71fd84f63127c9214e3c9422a651042e8016a94c2379ed368a57f2b7d not found: ID does not exist" Nov 01 22:27:53.624334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.624278 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" podStartSLOduration=2.142482688 podStartE2EDuration="4.624258775s" totalImagesPullingTime="2.481776087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:53.609837427 +0000 UTC m=+277.394276348" watchObservedRunningTime="2026-11-01 22:27:53.624258775 +0000 UTC m=+277.408697700" Nov 01 22:27:53.624529 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.624511 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt"] Nov 01 22:27:53.630551 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:53.630526 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt"] Nov 01 22:27:54.340311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:54.340272 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c027922-4282-41de-b78c-a1a1f5f129b2" path="/var/lib/kubelet/pods/1c027922-4282-41de-b78c-a1a1f5f129b2/volumes" Nov 01 22:27:54.597015 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:54.596923 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" event={"ID":"51eead1c-88d4-4e16-b360-6f370b3041b8","Type":"ContainerStarted","Data":"b74f82a6da0b1a78f744c7bc3cb7092073b45b74b75ac45859db7da19d4ce874"} Nov 01 22:27:54.597449 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:54.597026 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" Nov 01 22:27:54.599337 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:54.599305 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-rs5t6" event={"ID":"b2094375-d4ac-4cf1-a8da-bb4e90ba110c","Type":"ContainerStarted","Data":"2e4cca8390eb17040289e197a3d50efaedbf2daa6a11f838815854a54d7dd05e"} Nov 01 22:27:54.633476 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:54.633404 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-rs5t6" podStartSLOduration=2.005241194 podStartE2EDuration="5.633386114s" totalImagesPullingTime="3.62814492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:54.632602136 +0000 UTC m=+278.417041069" watchObservedRunningTime="2026-11-01 22:27:54.633386114 +0000 UTC m=+278.417825032" Nov 01 22:27:54.634258 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:54.634225 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" podStartSLOduration=2.126866547 podStartE2EDuration="5.634217208s" totalImagesPullingTime="3.507350661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:54.612927275 +0000 UTC m=+278.397366196" watchObservedRunningTime="2026-11-01 22:27:54.634217208 +0000 UTC m=+278.418656129" Nov 01 22:27:55.604153 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:55.604122 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:27:57.924880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:57.924787 6466 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-fqtxt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.18:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:27:57.924880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:27:57.924857 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fqtxt" podUID="1c027922-4282-41de-b78c-a1a1f5f129b2" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.18:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:28:04.604052 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:04.604023 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" Nov 01 22:28:06.610658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:06.610629 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-rs5t6" Nov 01 22:28:19.280252 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:28:19.280208 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:28:22.515612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.515580 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9"] Nov 01 22:28:22.516025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.515974 6466 cpu_manager.go:414] "containerMap: removing container" podUID="1c027922-4282-41de-b78c-a1a1f5f129b2" containerName="discovery" Nov 01 22:28:22.516025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.515986 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c027922-4282-41de-b78c-a1a1f5f129b2" containerName="discovery" Nov 01 22:28:22.516135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.516058 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c027922-4282-41de-b78c-a1a1f5f129b2" containerName="discovery" Nov 01 22:28:22.575503 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.575465 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9"] Nov 01 22:28:22.575667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.575568 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.580431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.580404 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 22:28:22.580581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.580453 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:28:22.580581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.580456 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:28:22.580581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.580492 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 22:28:22.580581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.580532 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7x7m\"" Nov 01 22:28:22.621714 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.621676 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-metrics-certs\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.621931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.621800 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.622010 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.621942 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-maas-api-tls\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.622010 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.621987 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzfdl\" (UniqueName: \"kubernetes.io/projected/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-kube-api-access-fzfdl\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.723003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.722969 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-metrics-certs\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.723003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.723002 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.723274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.723158 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-maas-api-tls\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.723274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.723203 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzfdl\" (UniqueName: \"kubernetes.io/projected/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-kube-api-access-fzfdl\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.723673 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.723641 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.725598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.725570 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-metrics-certs\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.725705 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.725615 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-maas-api-tls\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.730971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.730948 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzfdl\" (UniqueName: \"kubernetes.io/projected/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-kube-api-access-fzfdl\") pod \"maas-api-6d669c7fcd-76lk9\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:22.905224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:22.905187 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:23.198233 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:23.198211 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9"] Nov 01 22:28:23.200971 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:28:23.200940 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1929cfb5_c2f8_4b4a_bb72_583b84b6e27e.slice/crio-448ee3db584c8ece250b4985aaa155af0aedfe395543fe63033c9644340f7ad9 WatchSource:0}: Error finding container 448ee3db584c8ece250b4985aaa155af0aedfe395543fe63033c9644340f7ad9: Status 404 returned error can't find the container with id 448ee3db584c8ece250b4985aaa155af0aedfe395543fe63033c9644340f7ad9 Nov 01 22:28:23.733873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:23.733836 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" event={"ID":"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e","Type":"ContainerStarted","Data":"448ee3db584c8ece250b4985aaa155af0aedfe395543fe63033c9644340f7ad9"} Nov 01 22:28:25.610443 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:25.610414 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ngj6m" Nov 01 22:28:26.748534 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:26.748496 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" event={"ID":"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e","Type":"ContainerStarted","Data":"aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b"} Nov 01 22:28:26.749018 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:26.748605 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:26.766999 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:26.766950 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" podStartSLOduration=2.207066492 podStartE2EDuration="4.766935707s" totalImagesPullingTime="2.559869215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:26.765901947 +0000 UTC m=+310.550340870" watchObservedRunningTime="2026-11-01 22:28:26.766935707 +0000 UTC m=+310.551374658" Nov 01 22:28:32.759577 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:32.759545 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:33.526727 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.526693 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-f8f4j"] Nov 01 22:28:33.530241 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.530220 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" Nov 01 22:28:33.537249 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.537224 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-f8f4j"] Nov 01 22:28:33.562045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.562020 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 22:28:33.625048 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.625016 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-tls-cert\") pod \"authorino-6889cbd8c4-f8f4j\" (UID: \"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25\") " pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" Nov 01 22:28:33.625048 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.625052 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgcdb\" (UniqueName: \"kubernetes.io/projected/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-kube-api-access-cgcdb\") pod \"authorino-6889cbd8c4-f8f4j\" (UID: \"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25\") " pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" Nov 01 22:28:33.726317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.726287 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-tls-cert\") pod \"authorino-6889cbd8c4-f8f4j\" (UID: \"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25\") " pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" Nov 01 22:28:33.726317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.726319 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgcdb\" (UniqueName: \"kubernetes.io/projected/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-kube-api-access-cgcdb\") pod \"authorino-6889cbd8c4-f8f4j\" (UID: \"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25\") " pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" Nov 01 22:28:33.728705 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.728672 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-tls-cert\") pod \"authorino-6889cbd8c4-f8f4j\" (UID: \"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25\") " pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" Nov 01 22:28:33.734076 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.734052 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgcdb\" (UniqueName: \"kubernetes.io/projected/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-kube-api-access-cgcdb\") pod \"authorino-6889cbd8c4-f8f4j\" (UID: \"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25\") " pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" Nov 01 22:28:33.803546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.803473 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-f8f4j"] Nov 01 22:28:33.803884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:33.803670 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" Nov 01 22:28:34.085459 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:34.085383 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-f8f4j"] Nov 01 22:28:34.088841 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:28:34.088812 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0c42e1a_4d7c_42c9_b4e5_2398daf11a25.slice/crio-e7b44891ffc9182a4dca28acf8cc1fbc07140920be50e1343911bb25fe2bf8cb WatchSource:0}: Error finding container e7b44891ffc9182a4dca28acf8cc1fbc07140920be50e1343911bb25fe2bf8cb: Status 404 returned error can't find the container with id e7b44891ffc9182a4dca28acf8cc1fbc07140920be50e1343911bb25fe2bf8cb Nov 01 22:28:34.784633 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:34.784595 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" event={"ID":"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25","Type":"ContainerStarted","Data":"e7b44891ffc9182a4dca28acf8cc1fbc07140920be50e1343911bb25fe2bf8cb"} Nov 01 22:28:35.789969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:35.789929 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" event={"ID":"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25","Type":"ContainerStarted","Data":"a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2"} Nov 01 22:28:35.789969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:35.789944 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" podUID="f0c42e1a-4d7c-42c9-b4e5-2398daf11a25" containerName="authorino" containerID="cri-o://a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2" gracePeriod=30 Nov 01 22:28:35.804840 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:35.804764 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" podStartSLOduration=1.8435003490000001 podStartE2EDuration="2.804744225s" totalImagesPullingTime="961.243876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:35.80409718 +0000 UTC m=+319.588536101" watchObservedRunningTime="2026-11-01 22:28:35.804744225 +0000 UTC m=+319.589183147" Nov 01 22:28:36.048552 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.048493 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" Nov 01 22:28:36.146574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.146542 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-tls-cert\" (UniqueName: \"kubernetes.io/secret/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-tls-cert\") pod \"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25\" (UID: \"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25\") " Nov 01 22:28:36.146775 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.146602 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-kube-api-access-cgcdb\" (UniqueName: \"kubernetes.io/projected/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-kube-api-access-cgcdb\") pod \"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25\" (UID: \"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25\") " Nov 01 22:28:36.148702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.148658 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-kube-api-access-cgcdb" pod "f0c42e1a-4d7c-42c9-b4e5-2398daf11a25" (UID: "f0c42e1a-4d7c-42c9-b4e5-2398daf11a25"). InnerVolumeSpecName "kube-api-access-cgcdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:36.158382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.158355 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-tls-cert" pod "f0c42e1a-4d7c-42c9-b4e5-2398daf11a25" (UID: "f0c42e1a-4d7c-42c9-b4e5-2398daf11a25"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:36.179512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.179473 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zxb9h"] Nov 01 22:28:36.179679 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.179648 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" podUID="4c693bf5-4a52-43c1-b33a-9450328312e9" containerName="authorino" containerID="cri-o://6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562" gracePeriod=30 Nov 01 22:28:36.247780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.247751 6466 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-tls-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:36.247780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.247777 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgcdb\" (UniqueName: \"kubernetes.io/projected/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25-kube-api-access-cgcdb\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:36.420125 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.420090 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" Nov 01 22:28:36.550722 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.550688 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c693bf5-4a52-43c1-b33a-9450328312e9-kube-api-access-tqht2\" (UniqueName: \"kubernetes.io/projected/4c693bf5-4a52-43c1-b33a-9450328312e9-kube-api-access-tqht2\") pod \"4c693bf5-4a52-43c1-b33a-9450328312e9\" (UID: \"4c693bf5-4a52-43c1-b33a-9450328312e9\") " Nov 01 22:28:36.552718 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.552694 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c693bf5-4a52-43c1-b33a-9450328312e9-kube-api-access-tqht2" pod "4c693bf5-4a52-43c1-b33a-9450328312e9" (UID: "4c693bf5-4a52-43c1-b33a-9450328312e9"). InnerVolumeSpecName "kube-api-access-tqht2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:36.651815 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.651783 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqht2\" (UniqueName: \"kubernetes.io/projected/4c693bf5-4a52-43c1-b33a-9450328312e9-kube-api-access-tqht2\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:36.795749 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.795714 6466 generic.go:356] "Generic (PLEG): container finished" podID="f0c42e1a-4d7c-42c9-b4e5-2398daf11a25" containerID="a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2" exitCode=0 Nov 01 22:28:36.796198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.795768 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" Nov 01 22:28:36.796198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.795797 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" event={"ID":"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25","Type":"ContainerDied","Data":"a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2"} Nov 01 22:28:36.796198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.795844 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-f8f4j" event={"ID":"f0c42e1a-4d7c-42c9-b4e5-2398daf11a25","Type":"ContainerDied","Data":"e7b44891ffc9182a4dca28acf8cc1fbc07140920be50e1343911bb25fe2bf8cb"} Nov 01 22:28:36.796198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.795861 6466 scope.go:122] "RemoveContainer" containerID="a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2" Nov 01 22:28:36.797223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.797197 6466 generic.go:356] "Generic (PLEG): container finished" podID="4c693bf5-4a52-43c1-b33a-9450328312e9" containerID="6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562" exitCode=0 Nov 01 22:28:36.797309 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.797244 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" Nov 01 22:28:36.797309 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.797269 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" event={"ID":"4c693bf5-4a52-43c1-b33a-9450328312e9","Type":"ContainerDied","Data":"6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562"} Nov 01 22:28:36.797309 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.797296 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zxb9h" event={"ID":"4c693bf5-4a52-43c1-b33a-9450328312e9","Type":"ContainerDied","Data":"eb8fbbb5e355630d138eda7dda713342ba7deea13c31c2e74ce816c2cd5dc351"} Nov 01 22:28:36.805022 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.804933 6466 scope.go:122] "RemoveContainer" containerID="a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2" Nov 01 22:28:36.805270 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:28:36.805245 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2\": container with ID starting with a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2 not found: ID does not exist" containerID="a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2" Nov 01 22:28:36.805322 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.805284 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2"} err="failed to get container status \"a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2\": rpc error: code = NotFound desc = could not find container \"a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2\": container with ID starting with a6d13e7d00a8ff4435fa8c143d7fa0559cf47e22584d62d6e6fd889718281ae2 not found: ID does not exist" Nov 01 22:28:36.805322 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.805302 6466 scope.go:122] "RemoveContainer" containerID="6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562" Nov 01 22:28:36.811872 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.811850 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-f8f4j"] Nov 01 22:28:36.813512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.813493 6466 scope.go:122] "RemoveContainer" containerID="6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562" Nov 01 22:28:36.813862 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:28:36.813787 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562\": container with ID starting with 6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562 not found: ID does not exist" containerID="6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562" Nov 01 22:28:36.813862 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.813832 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562"} err="failed to get container status \"6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562\": rpc error: code = NotFound desc = could not find container \"6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562\": container with ID starting with 6be6315347a3b4b8ad40e82ea3695b9de604fa7406e7174a602474a0f2057562 not found: ID does not exist" Nov 01 22:28:36.815022 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.815004 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-f8f4j"] Nov 01 22:28:36.830330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.830309 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zxb9h"] Nov 01 22:28:36.835379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:36.835359 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zxb9h"] Nov 01 22:28:38.338564 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:38.338530 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c693bf5-4a52-43c1-b33a-9450328312e9" path="/var/lib/kubelet/pods/4c693bf5-4a52-43c1-b33a-9450328312e9/volumes" Nov 01 22:28:38.338945 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:38.338884 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c42e1a-4d7c-42c9-b4e5-2398daf11a25" path="/var/lib/kubelet/pods/f0c42e1a-4d7c-42c9-b4e5-2398daf11a25/volumes" Nov 01 22:28:42.349713 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.349684 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9"] Nov 01 22:28:42.350181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.349882 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" podUID="1929cfb5-c2f8-4b4a-bb72-583b84b6e27e" containerName="maas-api" containerID="cri-o://aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b" gracePeriod=30 Nov 01 22:28:42.594972 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.594946 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:42.702171 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.702132 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-metrics-certs\") pod \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " Nov 01 22:28:42.702171 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.702172 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-maas-api-tls\") pod \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " Nov 01 22:28:42.702429 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.702210 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-openshift-service-ca\") pod \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " Nov 01 22:28:42.702429 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.702301 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-kube-api-access-fzfdl\" (UniqueName: \"kubernetes.io/projected/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-kube-api-access-fzfdl\") pod \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\" (UID: \"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e\") " Nov 01 22:28:42.702622 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.702596 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-openshift-service-ca" pod "1929cfb5-c2f8-4b4a-bb72-583b84b6e27e" (UID: "1929cfb5-c2f8-4b4a-bb72-583b84b6e27e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:42.704488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.704454 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-maas-api-tls" pod "1929cfb5-c2f8-4b4a-bb72-583b84b6e27e" (UID: "1929cfb5-c2f8-4b4a-bb72-583b84b6e27e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:42.704612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.704561 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-kube-api-access-fzfdl" pod "1929cfb5-c2f8-4b4a-bb72-583b84b6e27e" (UID: "1929cfb5-c2f8-4b4a-bb72-583b84b6e27e"). InnerVolumeSpecName "kube-api-access-fzfdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:42.704612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.704568 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-metrics-certs" pod "1929cfb5-c2f8-4b4a-bb72-583b84b6e27e" (UID: "1929cfb5-c2f8-4b4a-bb72-583b84b6e27e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:42.803587 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.803544 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:42.803587 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.803579 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:42.803587 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.803593 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:42.803832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.803605 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzfdl\" (UniqueName: \"kubernetes.io/projected/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e-kube-api-access-fzfdl\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:42.826454 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.826423 6466 generic.go:356] "Generic (PLEG): container finished" podID="1929cfb5-c2f8-4b4a-bb72-583b84b6e27e" containerID="aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b" exitCode=0 Nov 01 22:28:42.826594 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.826472 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" Nov 01 22:28:42.826594 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.826479 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" event={"ID":"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e","Type":"ContainerDied","Data":"aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b"} Nov 01 22:28:42.826594 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.826521 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9" event={"ID":"1929cfb5-c2f8-4b4a-bb72-583b84b6e27e","Type":"ContainerDied","Data":"448ee3db584c8ece250b4985aaa155af0aedfe395543fe63033c9644340f7ad9"} Nov 01 22:28:42.826594 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.826542 6466 scope.go:122] "RemoveContainer" containerID="aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b" Nov 01 22:28:42.839058 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.837082 6466 scope.go:122] "RemoveContainer" containerID="aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b" Nov 01 22:28:42.839058 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:28:42.837459 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b\": container with ID starting with aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b not found: ID does not exist" containerID="aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b" Nov 01 22:28:42.839058 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.837500 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b"} err="failed to get container status \"aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b\": rpc error: code = NotFound desc = could not find container \"aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b\": container with ID starting with aaec91c903790f5e8e1b86276a9c73a9165ecde43dac83152489937b03fd5c3b not found: ID does not exist" Nov 01 22:28:42.853411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.853377 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9"] Nov 01 22:28:42.859976 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:42.859953 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-76lk9"] Nov 01 22:28:44.339464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:28:44.339425 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1929cfb5-c2f8-4b4a-bb72-583b84b6e27e" path="/var/lib/kubelet/pods/1929cfb5-c2f8-4b4a-bb72-583b84b6e27e/volumes" Nov 01 22:28:49.280650 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:28:49.280612 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:29:19.279316 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:29:19.279281 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:29:49.279875 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:29:49.279830 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:29:51.940089 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.940057 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h"] Nov 01 22:29:51.940474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.940443 6466 cpu_manager.go:414] "containerMap: removing container" podUID="4c693bf5-4a52-43c1-b33a-9450328312e9" containerName="authorino" Nov 01 22:29:51.940474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.940456 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c693bf5-4a52-43c1-b33a-9450328312e9" containerName="authorino" Nov 01 22:29:51.940547 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.940478 6466 cpu_manager.go:414] "containerMap: removing container" podUID="1929cfb5-c2f8-4b4a-bb72-583b84b6e27e" containerName="maas-api" Nov 01 22:29:51.940547 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.940483 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1929cfb5-c2f8-4b4a-bb72-583b84b6e27e" containerName="maas-api" Nov 01 22:29:51.940547 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.940490 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f0c42e1a-4d7c-42c9-b4e5-2398daf11a25" containerName="authorino" Nov 01 22:29:51.940547 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.940495 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c42e1a-4d7c-42c9-b4e5-2398daf11a25" containerName="authorino" Nov 01 22:29:51.940667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.940555 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c693bf5-4a52-43c1-b33a-9450328312e9" containerName="authorino" Nov 01 22:29:51.940667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.940565 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="1929cfb5-c2f8-4b4a-bb72-583b84b6e27e" containerName="maas-api" Nov 01 22:29:51.940667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.940571 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c42e1a-4d7c-42c9-b4e5-2398daf11a25" containerName="authorino" Nov 01 22:29:51.943669 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.943643 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:51.952490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.952471 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 22:29:51.952774 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.952740 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 01 22:29:51.952774 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.952763 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 22:29:51.953864 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.953844 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wtxww\"" Nov 01 22:29:51.959278 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:51.959257 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h"] Nov 01 22:29:52.116070 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.116037 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmfm4\" (UniqueName: \"kubernetes.io/projected/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-kube-api-access-vmfm4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.116247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.116101 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.116247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.116202 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.116247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.116239 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.116351 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.116282 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.116391 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.116359 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.217290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.217209 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.217290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.217249 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.217488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.217297 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmfm4\" (UniqueName: \"kubernetes.io/projected/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-kube-api-access-vmfm4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.217488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.217343 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.217488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.217362 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.217488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.217383 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.217777 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.217746 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.217897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.217781 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.217897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.217827 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.219643 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.219616 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.219814 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.219795 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.225714 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.225691 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmfm4\" (UniqueName: \"kubernetes.io/projected/ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e-kube-api-access-vmfm4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h\" (UID: \"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.257595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.257570 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:52.534076 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.534046 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h"] Nov 01 22:29:52.535443 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:29:52.535417 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff725c3c_3a7c_41b6_9efb_6ce0ea30ef9e.slice/crio-6f388b73b925e0ae3ab3dd67c7e14ac6b607c7432acd730d67b2b2b01307ea05 WatchSource:0}: Error finding container 6f388b73b925e0ae3ab3dd67c7e14ac6b607c7432acd730d67b2b2b01307ea05: Status 404 returned error can't find the container with id 6f388b73b925e0ae3ab3dd67c7e14ac6b607c7432acd730d67b2b2b01307ea05 Nov 01 22:29:52.806393 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.806320 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7klkb"] Nov 01 22:29:52.811101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.811074 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:52.819088 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.819065 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 01 22:29:52.837952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.837919 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7klkb"] Nov 01 22:29:52.923398 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.923363 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:52.923539 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.923484 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24x2k\" (UniqueName: \"kubernetes.io/projected/66937625-7220-44c9-b11a-dbc627d0219c-kube-api-access-24x2k\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:52.923539 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.923525 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-home\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:52.923660 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.923567 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:52.923660 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.923590 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:52.923748 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:52.923729 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66937625-7220-44c9-b11a-dbc627d0219c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.024473 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.024434 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24x2k\" (UniqueName: \"kubernetes.io/projected/66937625-7220-44c9-b11a-dbc627d0219c-kube-api-access-24x2k\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.025005 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.024490 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-home\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.025005 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.024535 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.025005 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.024569 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.025005 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.024610 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66937625-7220-44c9-b11a-dbc627d0219c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.025005 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.024636 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.025319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.025059 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.025649 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.025615 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-home\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.025858 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.025840 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.028821 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.028796 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66937625-7220-44c9-b11a-dbc627d0219c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.029719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.029692 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66937625-7220-44c9-b11a-dbc627d0219c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.035702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.035679 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24x2k\" (UniqueName: \"kubernetes.io/projected/66937625-7220-44c9-b11a-dbc627d0219c-kube-api-access-24x2k\") pod \"echo-simulator-kserve-64b6f8f96c-7klkb\" (UID: \"66937625-7220-44c9-b11a-dbc627d0219c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.124666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.124580 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:53.145740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.145697 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" event={"ID":"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e","Type":"ContainerStarted","Data":"6f388b73b925e0ae3ab3dd67c7e14ac6b607c7432acd730d67b2b2b01307ea05"} Nov 01 22:29:53.415271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:53.415224 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7klkb"] Nov 01 22:29:53.418431 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:29:53.418399 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66937625_7220_44c9_b11a_dbc627d0219c.slice/crio-36a14e3ab556bd27ce3fbca88fd997d7ade35bb0f831ce272d30e5127caf8771 WatchSource:0}: Error finding container 36a14e3ab556bd27ce3fbca88fd997d7ade35bb0f831ce272d30e5127caf8771: Status 404 returned error can't find the container with id 36a14e3ab556bd27ce3fbca88fd997d7ade35bb0f831ce272d30e5127caf8771 Nov 01 22:29:54.152607 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.152567 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" event={"ID":"66937625-7220-44c9-b11a-dbc627d0219c","Type":"ContainerStarted","Data":"36a14e3ab556bd27ce3fbca88fd997d7ade35bb0f831ce272d30e5127caf8771"} Nov 01 22:29:54.154395 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.154368 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" event={"ID":"ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e","Type":"ContainerStarted","Data":"4637e69087a56d6f9e0dbd346ae67b0d6763d04ca16077ea5d0d7b01d6d9efaa"} Nov 01 22:29:54.173052 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.173003 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" podStartSLOduration=1.768160553 podStartE2EDuration="3.172988065s" totalImagesPullingTime="1.404827512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:54.172384771 +0000 UTC m=+397.956823691" watchObservedRunningTime="2026-11-01 22:29:54.172988065 +0000 UTC m=+397.957426980" Nov 01 22:29:54.246006 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.245350 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh"] Nov 01 22:29:54.265412 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.264754 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:29:54.265412 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.264799 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh"] Nov 01 22:29:54.265412 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.264896 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.270037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.269832 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 01 22:29:54.438583 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.438549 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.438583 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.438587 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.438811 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.438717 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7006c77f-be7e-43be-b47d-7aaef13e5cc6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.438811 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.438766 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.438811 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.438787 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.438904 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.438870 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjz6b\" (UniqueName: \"kubernetes.io/projected/7006c77f-be7e-43be-b47d-7aaef13e5cc6-kube-api-access-gjz6b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.539972 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.539887 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjz6b\" (UniqueName: \"kubernetes.io/projected/7006c77f-be7e-43be-b47d-7aaef13e5cc6-kube-api-access-gjz6b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.539972 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.539953 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.540227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.539977 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.540227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.540141 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7006c77f-be7e-43be-b47d-7aaef13e5cc6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.540227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.540187 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.540227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.540212 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.540515 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.540493 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.540587 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.540541 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.540639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.540594 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.542416 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.542389 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7006c77f-be7e-43be-b47d-7aaef13e5cc6-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.542871 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.542852 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7006c77f-be7e-43be-b47d-7aaef13e5cc6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.548169 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.548144 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjz6b\" (UniqueName: \"kubernetes.io/projected/7006c77f-be7e-43be-b47d-7aaef13e5cc6-kube-api-access-gjz6b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh\" (UID: \"7006c77f-be7e-43be-b47d-7aaef13e5cc6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.590068 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.590040 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:54.879748 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:54.879663 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh"] Nov 01 22:29:54.883658 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:29:54.883620 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7006c77f_be7e_43be_b47d_7aaef13e5cc6.slice/crio-42667da3c4efa6248de6f16212e60ad80e413331065c36448e8b22d29a2e629c WatchSource:0}: Error finding container 42667da3c4efa6248de6f16212e60ad80e413331065c36448e8b22d29a2e629c: Status 404 returned error can't find the container with id 42667da3c4efa6248de6f16212e60ad80e413331065c36448e8b22d29a2e629c Nov 01 22:29:55.161892 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:55.161861 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" event={"ID":"7006c77f-be7e-43be-b47d-7aaef13e5cc6","Type":"ContainerStarted","Data":"42667da3c4efa6248de6f16212e60ad80e413331065c36448e8b22d29a2e629c"} Nov 01 22:29:56.167312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.167273 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" event={"ID":"7006c77f-be7e-43be-b47d-7aaef13e5cc6","Type":"ContainerStarted","Data":"8d1a0cfcd3323f23c65a9477bc6d2a09ae9a325de7417b9aa0b3181041585521"} Nov 01 22:29:56.168836 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.168808 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" event={"ID":"66937625-7220-44c9-b11a-dbc627d0219c","Type":"ContainerStarted","Data":"9a58ab69f16c33aa2a521b26e5c6eb49635c3a3fc72887c4e2fcceac8dd1c985"} Nov 01 22:29:56.205578 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.205541 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p"] Nov 01 22:29:56.209177 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.209102 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" podStartSLOduration=1.93754398 podStartE2EDuration="2.209087935s" totalImagesPullingTime="271.543955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:56.207637173 +0000 UTC m=+399.992076094" watchObservedRunningTime="2026-11-01 22:29:56.209087935 +0000 UTC m=+399.993526857" Nov 01 22:29:56.209504 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.209485 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.225935 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.225910 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p"] Nov 01 22:29:56.229915 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.229895 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 01 22:29:56.251412 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.251386 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.251523 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.251509 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.251649 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.251622 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.251690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.251676 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.251726 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.251704 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnwqh\" (UniqueName: \"kubernetes.io/projected/20f8556e-bad5-456f-ad1a-14ef9a2ea949-kube-api-access-fnwqh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.252031 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.252007 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f8556e-bad5-456f-ad1a-14ef9a2ea949-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.267771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.267721 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" podStartSLOduration=2.584986881 podStartE2EDuration="4.267709338s" totalImagesPullingTime="1.682722457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:56.266721755 +0000 UTC m=+400.051160675" watchObservedRunningTime="2026-11-01 22:29:56.267709338 +0000 UTC m=+400.052148275" Nov 01 22:29:56.353035 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.353004 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.353035 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.353036 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.353275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.353075 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.353275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.353095 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.353275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.353123 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnwqh\" (UniqueName: \"kubernetes.io/projected/20f8556e-bad5-456f-ad1a-14ef9a2ea949-kube-api-access-fnwqh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.353275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.353162 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f8556e-bad5-456f-ad1a-14ef9a2ea949-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.353636 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.353594 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.353821 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.353797 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.353906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.353808 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.355368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.355348 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f8556e-bad5-456f-ad1a-14ef9a2ea949-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.355639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.355623 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f8556e-bad5-456f-ad1a-14ef9a2ea949-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.362861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.362841 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnwqh\" (UniqueName: \"kubernetes.io/projected/20f8556e-bad5-456f-ad1a-14ef9a2ea949-kube-api-access-fnwqh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5758p\" (UID: \"20f8556e-bad5-456f-ad1a-14ef9a2ea949\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.522429 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.522355 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:56.590435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.590403 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:29:56.799882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:56.799854 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p"] Nov 01 22:29:56.802130 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:29:56.802080 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f8556e_bad5_456f_ad1a_14ef9a2ea949.slice/crio-f56a01fe8811a81caf09d803ba6495812a80b79b613e3f8233124c84dd58de2c WatchSource:0}: Error finding container f56a01fe8811a81caf09d803ba6495812a80b79b613e3f8233124c84dd58de2c: Status 404 returned error can't find the container with id f56a01fe8811a81caf09d803ba6495812a80b79b613e3f8233124c84dd58de2c Nov 01 22:29:57.125064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.125030 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:29:57.176323 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.176286 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" event={"ID":"20f8556e-bad5-456f-ad1a-14ef9a2ea949","Type":"ContainerStarted","Data":"c0103172df4a640c9a4c5bffce198fe1685e816371fb58d3eb0065050206ea52"} Nov 01 22:29:57.176777 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.176341 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" event={"ID":"20f8556e-bad5-456f-ad1a-14ef9a2ea949","Type":"ContainerStarted","Data":"f56a01fe8811a81caf09d803ba6495812a80b79b613e3f8233124c84dd58de2c"} Nov 01 22:29:57.197731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.197673 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" podStartSLOduration=0.923413276 podStartE2EDuration="1.197655642s" totalImagesPullingTime="274.242366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:57.196482943 +0000 UTC m=+400.980921864" watchObservedRunningTime="2026-11-01 22:29:57.197655642 +0000 UTC m=+400.982094565" Nov 01 22:29:57.523188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.523162 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:29:57.883850 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.883771 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz"] Nov 01 22:29:57.890596 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.890559 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:57.901078 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.901046 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz"] Nov 01 22:29:57.906054 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.906029 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 01 22:29:57.966579 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.966546 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:57.966579 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.966584 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58de7f57-d10e-42ff-ab3e-3952b601bfce-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:57.966788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.966686 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:57.966788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.966718 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv2m6\" (UniqueName: \"kubernetes.io/projected/58de7f57-d10e-42ff-ab3e-3952b601bfce-kube-api-access-kv2m6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:57.966788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.966746 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:57.966888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:57.966857 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.067646 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.067612 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.067829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.067654 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv2m6\" (UniqueName: \"kubernetes.io/projected/58de7f57-d10e-42ff-ab3e-3952b601bfce-kube-api-access-kv2m6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.067829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.067682 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.067829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.067716 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.067829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.067760 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.067829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.067777 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58de7f57-d10e-42ff-ab3e-3952b601bfce-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.068152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.068132 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.068232 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.068212 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.068282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.068230 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.069909 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.069892 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58de7f57-d10e-42ff-ab3e-3952b601bfce-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.070280 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.070264 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58de7f57-d10e-42ff-ab3e-3952b601bfce-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.076021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.076006 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv2m6\" (UniqueName: \"kubernetes.io/projected/58de7f57-d10e-42ff-ab3e-3952b601bfce-kube-api-access-kv2m6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qkfnz\" (UID: \"58de7f57-d10e-42ff-ab3e-3952b601bfce\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.206259 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.206232 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:58.479930 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:58.479801 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz"] Nov 01 22:29:58.482545 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:29:58.482517 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58de7f57_d10e_42ff_ab3e_3952b601bfce.slice/crio-bb7bfbc17de902f6cbd8c47810d7d6278b48bac85e247e8166bef48dbeedb6af WatchSource:0}: Error finding container bb7bfbc17de902f6cbd8c47810d7d6278b48bac85e247e8166bef48dbeedb6af: Status 404 returned error can't find the container with id bb7bfbc17de902f6cbd8c47810d7d6278b48bac85e247e8166bef48dbeedb6af Nov 01 22:29:59.187070 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.187030 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" event={"ID":"58de7f57-d10e-42ff-ab3e-3952b601bfce","Type":"ContainerStarted","Data":"3ecc0955cb22fab66da75aa48dd1f196561055cdf702ca9bad71cf0227b67670"} Nov 01 22:29:59.187277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.187092 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" event={"ID":"58de7f57-d10e-42ff-ab3e-3952b601bfce","Type":"ContainerStarted","Data":"bb7bfbc17de902f6cbd8c47810d7d6278b48bac85e247e8166bef48dbeedb6af"} Nov 01 22:29:59.207539 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.207502 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5"] Nov 01 22:29:59.211879 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.211856 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:29:59.212045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.211944 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.214478 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.214433 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" podStartSLOduration=1.929381889 podStartE2EDuration="2.214420683s" totalImagesPullingTime="285.038794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:59.213088395 +0000 UTC m=+402.997527317" watchObservedRunningTime="2026-11-01 22:29:59.214420683 +0000 UTC m=+402.998859605" Nov 01 22:29:59.217895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.217875 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 01 22:29:59.223436 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.223407 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5"] Nov 01 22:29:59.278192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.278156 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.278386 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.278294 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.278461 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.278390 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.278521 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.278468 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4fv8\" (UniqueName: \"kubernetes.io/projected/ce3f4a45-aa4a-465f-a436-9c0f4a542380-kube-api-access-m4fv8\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.278521 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.278496 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3f4a45-aa4a-465f-a436-9c0f4a542380-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.278521 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.278517 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.379034 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.378996 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.379257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.379058 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.379257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.379091 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.379753 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.379471 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4fv8\" (UniqueName: \"kubernetes.io/projected/ce3f4a45-aa4a-465f-a436-9c0f4a542380-kube-api-access-m4fv8\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.379753 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.379532 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.379753 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.379550 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3f4a45-aa4a-465f-a436-9c0f4a542380-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.379753 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.379578 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.379753 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.379579 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.380004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.379901 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.381981 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.381960 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3f4a45-aa4a-465f-a436-9c0f4a542380-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.382168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.382149 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3f4a45-aa4a-465f-a436-9c0f4a542380-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.390901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.390881 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4fv8\" (UniqueName: \"kubernetes.io/projected/ce3f4a45-aa4a-465f-a436-9c0f4a542380-kube-api-access-m4fv8\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lxpj5\" (UID: \"ce3f4a45-aa4a-465f-a436-9c0f4a542380\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.528732 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.528642 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:29:59.800020 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:29:59.799944 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5"] Nov 01 22:29:59.802046 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:29:59.802019 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3f4a45_aa4a_465f_a436_9c0f4a542380.slice/crio-dc145ac773c179d0327c2aecbcb4608d5599339f6c0eefa5adc0feae1d0caeb0 WatchSource:0}: Error finding container dc145ac773c179d0327c2aecbcb4608d5599339f6c0eefa5adc0feae1d0caeb0: Status 404 returned error can't find the container with id dc145ac773c179d0327c2aecbcb4608d5599339f6c0eefa5adc0feae1d0caeb0 Nov 01 22:30:00.193685 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:00.193659 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" event={"ID":"ce3f4a45-aa4a-465f-a436-9c0f4a542380","Type":"ContainerStarted","Data":"dc145ac773c179d0327c2aecbcb4608d5599339f6c0eefa5adc0feae1d0caeb0"} Nov 01 22:30:01.199868 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:01.199826 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" event={"ID":"ce3f4a45-aa4a-465f-a436-9c0f4a542380","Type":"ContainerStarted","Data":"0e412ccf52bcdd56e514fa9af632e744ef1e791e6e199dc68b7d8f2968c2f2b8"} Nov 01 22:30:01.529363 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:01.529269 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:30:02.258688 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:02.258648 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:30:02.271427 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:02.271398 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:30:02.292245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:02.292193 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" podStartSLOduration=2.961124913 podStartE2EDuration="3.292177816s" totalImagesPullingTime="331.052903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:01.217867718 +0000 UTC m=+405.002306639" watchObservedRunningTime="2026-11-01 22:30:02.292177816 +0000 UTC m=+406.076616738" Nov 01 22:30:03.125651 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:03.125606 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:30:03.138680 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:03.138648 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:30:03.219577 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:03.219546 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" Nov 01 22:30:03.220391 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:03.220373 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" Nov 01 22:30:04.590833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:04.590794 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:30:04.603789 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:04.603767 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:30:05.229091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:05.229062 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:30:06.522911 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:06.522876 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:30:06.535619 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:06.535595 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:30:07.239062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:07.239031 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" Nov 01 22:30:08.207397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:08.207360 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:30:08.220298 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:08.220266 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:30:08.245766 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:08.245739 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" Nov 01 22:30:09.529797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:09.529761 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:30:09.542390 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:09.542365 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:30:10.254832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:10.254796 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" Nov 01 22:30:19.280726 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:30:19.280680 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:30:19.685996 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:19.685961 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv"] Nov 01 22:30:19.691179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:19.691156 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" Nov 01 22:30:19.704296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:19.704268 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nnlzm\"" Nov 01 22:30:19.704296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:19.704275 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 22:30:19.704936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:19.704913 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv"] Nov 01 22:30:19.785311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:19.785277 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c96ea8-9914-43f7-a349-2a8322a7839c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rvxmv\" (UID: \"96c96ea8-9914-43f7-a349-2a8322a7839c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" Nov 01 22:30:19.785559 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:19.785340 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zb9z\" (UniqueName: \"kubernetes.io/projected/96c96ea8-9914-43f7-a349-2a8322a7839c-kube-api-access-6zb9z\") pod \"payload-pre-processing-7b98bb678c-rvxmv\" (UID: \"96c96ea8-9914-43f7-a349-2a8322a7839c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" Nov 01 22:30:19.885924 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:19.885885 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c96ea8-9914-43f7-a349-2a8322a7839c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rvxmv\" (UID: \"96c96ea8-9914-43f7-a349-2a8322a7839c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" Nov 01 22:30:19.886091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:19.885946 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zb9z\" (UniqueName: \"kubernetes.io/projected/96c96ea8-9914-43f7-a349-2a8322a7839c-kube-api-access-6zb9z\") pod \"payload-pre-processing-7b98bb678c-rvxmv\" (UID: \"96c96ea8-9914-43f7-a349-2a8322a7839c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" Nov 01 22:30:19.886744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:19.886726 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c96ea8-9914-43f7-a349-2a8322a7839c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rvxmv\" (UID: \"96c96ea8-9914-43f7-a349-2a8322a7839c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" Nov 01 22:30:19.894092 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:19.894062 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zb9z\" (UniqueName: \"kubernetes.io/projected/96c96ea8-9914-43f7-a349-2a8322a7839c-kube-api-access-6zb9z\") pod \"payload-pre-processing-7b98bb678c-rvxmv\" (UID: \"96c96ea8-9914-43f7-a349-2a8322a7839c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" Nov 01 22:30:20.005708 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:20.005623 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" Nov 01 22:30:20.278814 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:20.278740 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv"] Nov 01 22:30:20.281852 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:30:20.281825 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c96ea8_9914_43f7_a349_2a8322a7839c.slice/crio-4faf5f9f2e0132ab1dd7fa5ce8cd267dbe3429c92e39d687c9ac7a39c857ca90 WatchSource:0}: Error finding container 4faf5f9f2e0132ab1dd7fa5ce8cd267dbe3429c92e39d687c9ac7a39c857ca90: Status 404 returned error can't find the container with id 4faf5f9f2e0132ab1dd7fa5ce8cd267dbe3429c92e39d687c9ac7a39c857ca90 Nov 01 22:30:20.287643 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:20.287610 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" event={"ID":"96c96ea8-9914-43f7-a349-2a8322a7839c","Type":"ContainerStarted","Data":"4faf5f9f2e0132ab1dd7fa5ce8cd267dbe3429c92e39d687c9ac7a39c857ca90"} Nov 01 22:30:22.514940 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.514911 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59"] Nov 01 22:30:22.520463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.520236 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.543817 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.543785 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59"] Nov 01 22:30:22.615872 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.615843 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.616009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.615877 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6mlr\" (UniqueName: \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-kube-api-access-p6mlr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.616009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.615900 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.616083 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.616052 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.616146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.616102 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.616282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.616263 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.616348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.616302 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.616348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.616336 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.616446 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.616373 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.616755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.616741 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-flhqn\"" Nov 01 22:30:22.717490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.717456 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.717702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.717512 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.717702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.717530 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.717702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.717551 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.717702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.717574 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.717702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.717602 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.717702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.717662 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mlr\" (UniqueName: \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-kube-api-access-p6mlr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.718021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.717711 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.718021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.717813 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.718021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.717963 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.718212 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.718151 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.718265 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.718230 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.718304 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.718267 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.718519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.718498 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.719892 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.719870 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.720179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.720158 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.725825 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.725799 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.726030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.726008 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6mlr\" (UniqueName: \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-kube-api-access-p6mlr\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9gn59\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:22.910886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:22.910849 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:23.189598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:23.189518 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59"] Nov 01 22:30:23.192461 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:30:23.192432 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9169aa9_28c9_4698_8d11_8b0d232d81a0.slice/crio-388699ed7aa5d39418600c8fc36e94303575ad4a968a31fcccbf68f03b6d39c5 WatchSource:0}: Error finding container 388699ed7aa5d39418600c8fc36e94303575ad4a968a31fcccbf68f03b6d39c5: Status 404 returned error can't find the container with id 388699ed7aa5d39418600c8fc36e94303575ad4a968a31fcccbf68f03b6d39c5 Nov 01 22:30:23.302889 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:23.302854 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" event={"ID":"b9169aa9-28c9-4698-8d11-8b0d232d81a0","Type":"ContainerStarted","Data":"388699ed7aa5d39418600c8fc36e94303575ad4a968a31fcccbf68f03b6d39c5"} Nov 01 22:30:23.304327 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:23.304300 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" event={"ID":"96c96ea8-9914-43f7-a349-2a8322a7839c","Type":"ContainerStarted","Data":"e5eaca053d9f15336f0114b20f467f9a7b88dc4e71c1399ad052f9b3086d83f2"} Nov 01 22:30:23.304468 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:23.304418 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" Nov 01 22:30:23.322204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:23.322144 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" podStartSLOduration=2.10124784 podStartE2EDuration="4.322127261s" totalImagesPullingTime="2.220879421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:23.320622293 +0000 UTC m=+427.105061239" watchObservedRunningTime="2026-11-01 22:30:23.322127261 +0000 UTC m=+427.106566183" Nov 01 22:30:32.510421 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:32.510390 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:30:32.510717 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:32.510455 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:30:32.510717 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:32.510486 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:30:33.365395 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:33.365360 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" event={"ID":"b9169aa9-28c9-4698-8d11-8b0d232d81a0","Type":"ContainerStarted","Data":"74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9"} Nov 01 22:30:33.388834 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:33.388783 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" podStartSLOduration=2.073986686 podStartE2EDuration="11.388770008s" totalImagesPullingTime="9.314783322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:33.387849817 +0000 UTC m=+437.172288740" watchObservedRunningTime="2026-11-01 22:30:33.388770008 +0000 UTC m=+437.173208928" Nov 01 22:30:33.911116 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:33.911065 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:33.912887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:33.912861 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9gn59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 22:30:33.913005 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:33.912910 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 22:30:34.312496 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:34.312413 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rvxmv" Nov 01 22:30:34.911692 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:34.911656 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9gn59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 22:30:34.912068 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:34.911714 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 22:30:35.911377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:35.911340 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9gn59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 22:30:35.911565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:35.911393 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 22:30:36.911611 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:36.911570 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9gn59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 22:30:36.912073 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:36.911639 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 22:30:37.912017 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:37.911979 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:37.912431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:37.912271 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9gn59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 22:30:37.912431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:37.912333 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 22:30:38.911842 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:38.911800 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9gn59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 22:30:38.912057 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:38.911881 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 22:30:39.911268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:39.911225 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9gn59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 22:30:39.911439 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:39.911294 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 22:30:40.915220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:40.915190 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:40.916227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:30:40.916204 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 22:30:49.280400 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:30:49.280363 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:31:13.571627 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.571542 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8"] Nov 01 22:31:13.580751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.580721 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.590814 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.590784 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b42cc4-openshift-default-dockercfg-lt2tw\"" Nov 01 22:31:13.591682 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.591654 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8"] Nov 01 22:31:13.741097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.741065 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zc7h\" (UniqueName: \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-kube-api-access-6zc7h\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.741294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.741104 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.741294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.741224 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.741396 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.741347 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.741433 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.741411 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.741485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.741430 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.741545 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.741490 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52a7224-6cd7-44ea-9bf5-d16246da3250-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.741622 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.741606 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.741660 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.741633 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.745394 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.745370 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn"] Nov 01 22:31:13.749551 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.749529 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:13.763960 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.763932 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn"] Nov 01 22:31:13.780067 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.780042 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-381659-openshift-default-dockercfg-wjt66\"" Nov 01 22:31:13.842642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.842568 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52a7224-6cd7-44ea-9bf5-d16246da3250-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.842642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.842618 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.842642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.842635 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.843003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.842690 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zc7h\" (UniqueName: \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-kube-api-access-6zc7h\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.843003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.842709 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.843003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.842734 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.843003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.842932 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.843234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.843015 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.843234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.843047 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.843234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.843130 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.843341 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.843237 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.843407 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.843385 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.843456 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.843412 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52a7224-6cd7-44ea-9bf5-d16246da3250-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.843456 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.843444 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.845189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.845102 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.845454 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.845430 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.858498 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.858459 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.858806 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.858781 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zc7h\" (UniqueName: \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-kube-api-access-6zc7h\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.898861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.898832 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:13.924721 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.924680 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx"] Nov 01 22:31:13.930486 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.930459 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:13.935783 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.935721 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3d8490-openshift-default-dockercfg-vp9kv\"" Nov 01 22:31:13.945210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.944543 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:13.945210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.944601 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:13.945210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.944632 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:13.945210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.944663 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:13.945210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.944730 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:13.945210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.944782 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:13.945210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.944816 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx"] Nov 01 22:31:13.945210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.944821 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdtqt\" (UniqueName: \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-kube-api-access-hdtqt\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:13.945210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.945024 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53147cfc-27a6-4f61-863a-732bfd063831-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:13.945210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:13.945074 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53147cfc-27a6-4f61-863a-732bfd063831-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.046394 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046352 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53147cfc-27a6-4f61-863a-732bfd063831-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.046624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046402 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.046624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046441 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.046624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046536 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53147cfc-27a6-4f61-863a-732bfd063831-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.046624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046576 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.046624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046612 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046657 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046688 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046714 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046738 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm857\" (UniqueName: \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-kube-api-access-fm857\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046774 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046848 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046882 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a12f352-2870-4eb9-992f-d1d12348f299-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046904 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.046961 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.047014 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdtqt\" (UniqueName: \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-kube-api-access-hdtqt\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.047079 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a12f352-2870-4eb9-992f-d1d12348f299-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.047125 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.047948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.047103 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.048707 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.048231 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.048707 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.048541 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.048707 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.048545 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.049213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.049191 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53147cfc-27a6-4f61-863a-732bfd063831-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.050515 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.050492 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.050902 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.050885 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53147cfc-27a6-4f61-863a-732bfd063831-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.061569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.061542 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.071722 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.071637 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdtqt\" (UniqueName: \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-kube-api-access-hdtqt\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-whtxn\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.086807 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.086780 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg"] Nov 01 22:31:14.092077 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.092052 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.108432 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.108407 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-b28a40-openshift-default-dockercfg-mwllr\"" Nov 01 22:31:14.109435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.109414 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg"] Nov 01 22:31:14.148949 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.148912 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.148949 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.148953 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm857\" (UniqueName: \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-kube-api-access-fm857\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.149230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.149019 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a12f352-2870-4eb9-992f-d1d12348f299-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.149230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.149044 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.149230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.149142 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a12f352-2870-4eb9-992f-d1d12348f299-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.149230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.149166 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.149449 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.149331 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.149449 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.149383 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.149545 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.149444 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.149545 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.149463 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.149724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.149707 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.150038 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.150013 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a12f352-2870-4eb9-992f-d1d12348f299-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.150213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.150030 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.150405 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.150386 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.151760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.151738 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a12f352-2870-4eb9-992f-d1d12348f299-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.151919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.151903 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.158546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.158518 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.158646 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.158520 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm857\" (UniqueName: \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-kube-api-access-fm857\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.204375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.204343 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8"] Nov 01 22:31:14.206707 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:31:14.206683 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52a7224_6cd7_44ea_9bf5_d16246da3250.slice/crio-56dca0a7fd3e338da4ba35245dedcd2118f0182d095d49a417538ed192c8ec95 WatchSource:0}: Error finding container 56dca0a7fd3e338da4ba35245dedcd2118f0182d095d49a417538ed192c8ec95: Status 404 returned error can't find the container with id 56dca0a7fd3e338da4ba35245dedcd2118f0182d095d49a417538ed192c8ec95 Nov 01 22:31:14.208941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.208904 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.209045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.209020 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.209120 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.209088 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.254330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.251216 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx7cf\" (UniqueName: \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-kube-api-access-dx7cf\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.254330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.251266 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.254330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.251305 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.254330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.251355 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.254330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.251410 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.254330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.251467 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8831591d-4318-4b8d-b6bc-9790a73d8c67-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.254330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.251503 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.254330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.251551 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.254330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.251576 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.254330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.252909 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:14.355301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.355229 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7cf\" (UniqueName: \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-kube-api-access-dx7cf\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.355301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.355280 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.355492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.355329 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.355492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.355369 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.355492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.355411 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.355492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.355462 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8831591d-4318-4b8d-b6bc-9790a73d8c67-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.355492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.355488 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.355738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.355518 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.355738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.355534 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.357331 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.356570 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.357331 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.357241 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.357331 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.357267 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8831591d-4318-4b8d-b6bc-9790a73d8c67-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.357606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.357483 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.357755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.357736 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.359333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.359309 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.362136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.362092 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.365138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.365071 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:14.372424 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.372398 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7cf\" (UniqueName: \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-kube-api-access-dx7cf\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.374060 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.374032 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.406470 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.406433 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:14.572501 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.572463 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" event={"ID":"d52a7224-6cd7-44ea-9bf5-d16246da3250","Type":"ContainerStarted","Data":"46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91"} Nov 01 22:31:14.572993 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.572530 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" event={"ID":"d52a7224-6cd7-44ea-9bf5-d16246da3250","Type":"ContainerStarted","Data":"56dca0a7fd3e338da4ba35245dedcd2118f0182d095d49a417538ed192c8ec95"} Nov 01 22:31:14.580575 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.580552 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx"] Nov 01 22:31:14.582087 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:31:14.582056 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a12f352_2870_4eb9_992f_d1d12348f299.slice/crio-ec2f18fea57c40a60e0257bc820f6e2d734dc18511fea4a416450c832d5f6f01 WatchSource:0}: Error finding container ec2f18fea57c40a60e0257bc820f6e2d734dc18511fea4a416450c832d5f6f01: Status 404 returned error can't find the container with id ec2f18fea57c40a60e0257bc820f6e2d734dc18511fea4a416450c832d5f6f01 Nov 01 22:31:14.584408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.584382 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.584474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.584443 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.584508 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.584472 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.606101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.606016 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" podStartSLOduration=1.606000822 podStartE2EDuration="1.606000822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:14.604292182 +0000 UTC m=+478.388731104" watchObservedRunningTime="2026-11-01 22:31:14.606000822 +0000 UTC m=+478.390439818" Nov 01 22:31:14.668282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.668004 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn"] Nov 01 22:31:14.670944 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:31:14.670904 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53147cfc_27a6_4f61_863a_732bfd063831.slice/crio-09d71a33dd18cd2d8d665f2e4a34496fe040287018f37bde1bd04130ae083e28 WatchSource:0}: Error finding container 09d71a33dd18cd2d8d665f2e4a34496fe040287018f37bde1bd04130ae083e28: Status 404 returned error can't find the container with id 09d71a33dd18cd2d8d665f2e4a34496fe040287018f37bde1bd04130ae083e28 Nov 01 22:31:14.673368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.673328 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.673474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.673393 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.673474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.673429 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.731507 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.731404 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg"] Nov 01 22:31:14.750397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.750217 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.750397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.750353 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.750535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.750437 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:14.899563 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:14.899531 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:15.579628 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:15.579594 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" event={"ID":"53147cfc-27a6-4f61-863a-732bfd063831","Type":"ContainerStarted","Data":"ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98"} Nov 01 22:31:15.580091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:15.579651 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" event={"ID":"53147cfc-27a6-4f61-863a-732bfd063831","Type":"ContainerStarted","Data":"09d71a33dd18cd2d8d665f2e4a34496fe040287018f37bde1bd04130ae083e28"} Nov 01 22:31:15.581377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:15.581346 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" event={"ID":"9a12f352-2870-4eb9-992f-d1d12348f299","Type":"ContainerStarted","Data":"bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8"} Nov 01 22:31:15.581493 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:15.581403 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" event={"ID":"9a12f352-2870-4eb9-992f-d1d12348f299","Type":"ContainerStarted","Data":"ec2f18fea57c40a60e0257bc820f6e2d734dc18511fea4a416450c832d5f6f01"} Nov 01 22:31:15.582946 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:15.582919 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" event={"ID":"8831591d-4318-4b8d-b6bc-9790a73d8c67","Type":"ContainerStarted","Data":"3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593"} Nov 01 22:31:15.583043 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:15.582969 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" event={"ID":"8831591d-4318-4b8d-b6bc-9790a73d8c67","Type":"ContainerStarted","Data":"3cb6181342526f275e73e4c6d76010330d9d6cc719cc8fff787b32d0ca74f4f8"} Nov 01 22:31:15.601836 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:15.601784 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" podStartSLOduration=2.601769085 podStartE2EDuration="2.601769085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:15.601306267 +0000 UTC m=+479.385745200" watchObservedRunningTime="2026-11-01 22:31:15.601769085 +0000 UTC m=+479.386208010" Nov 01 22:31:15.624669 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:15.624619 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" podStartSLOduration=2.624605906 podStartE2EDuration="2.624605906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:15.622543734 +0000 UTC m=+479.406982669" watchObservedRunningTime="2026-11-01 22:31:15.624605906 +0000 UTC m=+479.409044826" Nov 01 22:31:15.644370 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:15.644315 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" podStartSLOduration=1.6442975899999999 podStartE2EDuration="1.64429759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:15.642724854 +0000 UTC m=+479.427163831" watchObservedRunningTime="2026-11-01 22:31:15.64429759 +0000 UTC m=+479.428736510" Nov 01 22:31:15.904781 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:15.904749 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:16.255063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.254987 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:16.260696 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.260664 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:16.365618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.365591 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:16.371365 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.371338 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:16.406860 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.406836 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:16.411881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.411858 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:16.589983 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.589890 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:16.589983 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.589946 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:16.590517 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.590350 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:16.590517 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.590427 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:16.590517 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.590482 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:16.590675 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.590529 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:16.590721 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.590704 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:16.590773 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:16.590763 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:19.020103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.020066 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k"] Nov 01 22:31:19.025658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.025630 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.038654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.037994 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k"] Nov 01 22:31:19.041844 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.041817 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-e25182-serving-cert\"" Nov 01 22:31:19.042041 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.042000 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7x7m\"" Nov 01 22:31:19.044243 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.044189 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 22:31:19.045945 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.045925 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:31:19.046081 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.045964 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:31:19.115032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.114993 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.115032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.115034 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-metrics-certs\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.115361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.115226 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bcccba9-871e-409f-80ce-269ca37ad8a0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.115361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.115265 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87dt\" (UniqueName: \"kubernetes.io/projected/0bcccba9-871e-409f-80ce-269ca37ad8a0-kube-api-access-z87dt\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.216881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.216839 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bcccba9-871e-409f-80ce-269ca37ad8a0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.217071 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.216896 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87dt\" (UniqueName: \"kubernetes.io/projected/0bcccba9-871e-409f-80ce-269ca37ad8a0-kube-api-access-z87dt\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.217071 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.216979 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.217071 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.217012 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-metrics-certs\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.217632 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.217602 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bcccba9-871e-409f-80ce-269ca37ad8a0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.221258 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.221234 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.221627 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.221603 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-metrics-certs\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.231895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.231873 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87dt\" (UniqueName: \"kubernetes.io/projected/0bcccba9-871e-409f-80ce-269ca37ad8a0-kube-api-access-z87dt\") pod \"maas-api-e2e-worker-w0-e25182-5f9775888b-t242k\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.315297 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:31:19.315213 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:31:19.346699 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.346668 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:19.630021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.629942 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k"] Nov 01 22:31:19.635072 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:31:19.635041 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcccba9_871e_409f_80ce_269ca37ad8a0.slice/crio-ba40882a4342acaf4f23336019b28648bef83ee946338bae79c1efed7bb29c39 WatchSource:0}: Error finding container ba40882a4342acaf4f23336019b28648bef83ee946338bae79c1efed7bb29c39: Status 404 returned error can't find the container with id ba40882a4342acaf4f23336019b28648bef83ee946338bae79c1efed7bb29c39 Nov 01 22:31:19.637017 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:19.636992 6466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:31:20.075541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.075518 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88"] Nov 01 22:31:20.079511 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.079478 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.093306 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.093286 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-00e49d-serving-cert\"" Nov 01 22:31:20.098759 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.098734 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88"] Nov 01 22:31:20.127451 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.127422 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-metrics-certs\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.127606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.127472 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-maas-api-tls\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.127606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.127495 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c704a06d-d1e6-4e05-8730-15cb9f7a4309-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.127606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.127592 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4tf\" (UniqueName: \"kubernetes.io/projected/c704a06d-d1e6-4e05-8730-15cb9f7a4309-kube-api-access-pj4tf\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.229276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.229230 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-metrics-certs\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.229473 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.229319 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-maas-api-tls\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.229473 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.229384 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c704a06d-d1e6-4e05-8730-15cb9f7a4309-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.229473 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.229431 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4tf\" (UniqueName: \"kubernetes.io/projected/c704a06d-d1e6-4e05-8730-15cb9f7a4309-kube-api-access-pj4tf\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.230623 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.230540 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c704a06d-d1e6-4e05-8730-15cb9f7a4309-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.233920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.233177 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-metrics-certs\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.233920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.233604 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-maas-api-tls\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.241129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.241083 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4tf\" (UniqueName: \"kubernetes.io/projected/c704a06d-d1e6-4e05-8730-15cb9f7a4309-kube-api-access-pj4tf\") pod \"maas-api-e2e-worker-w3-00e49d-597594b94-rqg88\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.394167 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.394131 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:20.613522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.613481 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" event={"ID":"0bcccba9-871e-409f-80ce-269ca37ad8a0","Type":"ContainerStarted","Data":"bd56549a3e312887f0a83072ba63d4ddcdeff4859a6dbb14385a368c952c00a8"} Nov 01 22:31:20.613709 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.613545 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" event={"ID":"0bcccba9-871e-409f-80ce-269ca37ad8a0","Type":"ContainerStarted","Data":"ba40882a4342acaf4f23336019b28648bef83ee946338bae79c1efed7bb29c39"} Nov 01 22:31:20.613709 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.613577 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:20.631603 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.631553 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" podStartSLOduration=2.246675919 podStartE2EDuration="2.631539487s" totalImagesPullingTime="384.863568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:20.630133844 +0000 UTC m=+484.414572767" watchObservedRunningTime="2026-11-01 22:31:20.631539487 +0000 UTC m=+484.415978408" Nov 01 22:31:20.668996 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.668972 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88"] Nov 01 22:31:20.671751 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:31:20.671720 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc704a06d_d1e6_4e05_8730_15cb9f7a4309.slice/crio-ba8ae3d9b1b19dacf07d38dbf703c823b0468ebc86a83958098e6e01ac115d60 WatchSource:0}: Error finding container ba8ae3d9b1b19dacf07d38dbf703c823b0468ebc86a83958098e6e01ac115d60: Status 404 returned error can't find the container with id ba8ae3d9b1b19dacf07d38dbf703c823b0468ebc86a83958098e6e01ac115d60 Nov 01 22:31:20.926872 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.926789 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t"] Nov 01 22:31:20.931783 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.931762 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:31:20.937789 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.937768 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-381659-dockercfg-ghtc7\"" Nov 01 22:31:20.947700 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.947668 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t"] Nov 01 22:31:20.996931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:20.996899 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-381659\"" Nov 01 22:31:21.050151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.044572 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t\" (UID: \"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:31:21.050151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.044643 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tslgv\" (UniqueName: \"kubernetes.io/projected/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-kube-api-access-tslgv\") pod \"payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t\" (UID: \"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:31:21.148205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.148175 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t\" (UID: \"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:31:21.148554 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.148231 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tslgv\" (UniqueName: \"kubernetes.io/projected/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-kube-api-access-tslgv\") pod \"payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t\" (UID: \"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:31:21.149035 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.149015 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t\" (UID: \"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:31:21.157530 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.157508 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tslgv\" (UniqueName: \"kubernetes.io/projected/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-kube-api-access-tslgv\") pod \"payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t\" (UID: \"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:31:21.250211 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.250091 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:31:21.550429 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.550272 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t"] Nov 01 22:31:21.553350 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:31:21.553286 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5fd4c8_80e4_4169_a6a7_30cbee6941b8.slice/crio-ca6ac8e36e4ee9f55c7ee45bfe5a703dd1fa55fc3ae09bb1e6fe5a99ad94e110 WatchSource:0}: Error finding container ca6ac8e36e4ee9f55c7ee45bfe5a703dd1fa55fc3ae09bb1e6fe5a99ad94e110: Status 404 returned error can't find the container with id ca6ac8e36e4ee9f55c7ee45bfe5a703dd1fa55fc3ae09bb1e6fe5a99ad94e110 Nov 01 22:31:21.620311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.620275 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" event={"ID":"c704a06d-d1e6-4e05-8730-15cb9f7a4309","Type":"ContainerStarted","Data":"4bd5f7d166c929b5699ae8b98a475ac56df9fd4fb26db76d1ee1f743f995a949"} Nov 01 22:31:21.620472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.620337 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" event={"ID":"c704a06d-d1e6-4e05-8730-15cb9f7a4309","Type":"ContainerStarted","Data":"ba8ae3d9b1b19dacf07d38dbf703c823b0468ebc86a83958098e6e01ac115d60"} Nov 01 22:31:21.620472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.620469 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:21.621728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.621704 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" event={"ID":"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8","Type":"ContainerStarted","Data":"ca6ac8e36e4ee9f55c7ee45bfe5a703dd1fa55fc3ae09bb1e6fe5a99ad94e110"} Nov 01 22:31:21.643971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.643917 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" podStartSLOduration=1.203599012 podStartE2EDuration="1.643904306s" totalImagesPullingTime="440.305294ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:21.643527636 +0000 UTC m=+485.427966558" watchObservedRunningTime="2026-11-01 22:31:21.643904306 +0000 UTC m=+485.428343264" Nov 01 22:31:21.834862 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.834788 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl"] Nov 01 22:31:21.838789 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.838769 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:31:21.846963 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.846936 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4"] Nov 01 22:31:21.849453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.849426 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-b28a40\"" Nov 01 22:31:21.851622 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.851602 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl"] Nov 01 22:31:21.851737 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.851693 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:31:21.858957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.858931 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-b28a40-dockercfg-rpxp6\"" Nov 01 22:31:21.868953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.868931 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4"] Nov 01 22:31:21.914092 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.914059 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l"] Nov 01 22:31:21.918064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.918038 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:21.930581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.930555 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l"] Nov 01 22:31:21.958491 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.958465 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-b42cc4-serving-cert\"" Nov 01 22:31:21.958805 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.958787 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-metrics-certs\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:21.958845 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.958822 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:21.958959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.958945 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl\" (UID: \"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:31:21.959006 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.958990 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jsf\" (UniqueName: \"kubernetes.io/projected/67394475-a855-440e-b6bd-8efb1ce7a539-kube-api-access-k9jsf\") pod \"payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4\" (UID: \"67394475-a855-440e-b6bd-8efb1ce7a539\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:31:21.959063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.959016 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73cb80d1-5953-4b69-9c6b-51eef06e8336-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:21.959170 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.959147 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6t5\" (UniqueName: \"kubernetes.io/projected/73cb80d1-5953-4b69-9c6b-51eef06e8336-kube-api-access-qx6t5\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:21.959274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.959259 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67394475-a855-440e-b6bd-8efb1ce7a539-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4\" (UID: \"67394475-a855-440e-b6bd-8efb1ce7a539\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:31:21.959348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:21.959331 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p2wx\" (UniqueName: \"kubernetes.io/projected/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-kube-api-access-8p2wx\") pod \"payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl\" (UID: \"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.059987 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl\" (UID: \"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.060081 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jsf\" (UniqueName: \"kubernetes.io/projected/67394475-a855-440e-b6bd-8efb1ce7a539-kube-api-access-k9jsf\") pod \"payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4\" (UID: \"67394475-a855-440e-b6bd-8efb1ce7a539\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.060142 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73cb80d1-5953-4b69-9c6b-51eef06e8336-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.060187 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6t5\" (UniqueName: \"kubernetes.io/projected/73cb80d1-5953-4b69-9c6b-51eef06e8336-kube-api-access-qx6t5\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.060275 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67394475-a855-440e-b6bd-8efb1ce7a539-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4\" (UID: \"67394475-a855-440e-b6bd-8efb1ce7a539\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.060316 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p2wx\" (UniqueName: \"kubernetes.io/projected/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-kube-api-access-8p2wx\") pod \"payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl\" (UID: \"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.060367 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-metrics-certs\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.060402 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:31:22.060584 6466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-serving-cert: secret "maas-api-e2e-worker-w4-b42cc4-serving-cert" not found Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:31:22.060673 6466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-maas-api-tls podName:73cb80d1-5953-4b69-9c6b-51eef06e8336 nodeName:}" failed. No retries permitted until 2026-11-01 22:31:22.560648855 +0000 UTC m=+486.345087774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-maas-api-tls") pod "maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" (UID: "73cb80d1-5953-4b69-9c6b-51eef06e8336") : secret "maas-api-e2e-worker-w4-b42cc4-serving-cert" not found Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.061663 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl\" (UID: \"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:31:22.063352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.062891 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73cb80d1-5953-4b69-9c6b-51eef06e8336-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:22.064865 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.064840 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67394475-a855-440e-b6bd-8efb1ce7a539-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4\" (UID: \"67394475-a855-440e-b6bd-8efb1ce7a539\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:31:22.069356 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.069333 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-metrics-certs\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:22.089057 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.088975 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p2wx\" (UniqueName: \"kubernetes.io/projected/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-kube-api-access-8p2wx\") pod \"payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl\" (UID: \"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:31:22.089576 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.089554 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jsf\" (UniqueName: \"kubernetes.io/projected/67394475-a855-440e-b6bd-8efb1ce7a539-kube-api-access-k9jsf\") pod \"payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4\" (UID: \"67394475-a855-440e-b6bd-8efb1ce7a539\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:31:22.092222 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.092187 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6t5\" (UniqueName: \"kubernetes.io/projected/73cb80d1-5953-4b69-9c6b-51eef06e8336-kube-api-access-qx6t5\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:22.154815 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.154777 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:31:22.168283 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.168255 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:31:22.440989 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.440966 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl"] Nov 01 22:31:22.443418 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:31:22.443380 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8a6a1e_70e8_4c0a_9a72_ca58a711841d.slice/crio-0195f83f74c3fbd555b202daf5cdc5973170412445bfe5c413e09c123ac7fae7 WatchSource:0}: Error finding container 0195f83f74c3fbd555b202daf5cdc5973170412445bfe5c413e09c123ac7fae7: Status 404 returned error can't find the container with id 0195f83f74c3fbd555b202daf5cdc5973170412445bfe5c413e09c123ac7fae7 Nov 01 22:31:22.478921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.478901 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4"] Nov 01 22:31:22.481168 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:31:22.481142 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67394475_a855_440e_b6bd_8efb1ce7a539.slice/crio-20a83b8c3b0e3c957d627a12a5c57cf74f60897eed847c1741bc5f783652e25b WatchSource:0}: Error finding container 20a83b8c3b0e3c957d627a12a5c57cf74f60897eed847c1741bc5f783652e25b: Status 404 returned error can't find the container with id 20a83b8c3b0e3c957d627a12a5c57cf74f60897eed847c1741bc5f783652e25b Nov 01 22:31:22.583050 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.578842 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:22.588546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.587684 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:22.628967 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.628933 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" event={"ID":"67394475-a855-440e-b6bd-8efb1ce7a539","Type":"ContainerStarted","Data":"e4bf276dd5d6b0ab312ecbd8178fa9235b8baa96bb27e1527bc1dd198a2deaec"} Nov 01 22:31:22.629162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.628999 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" event={"ID":"67394475-a855-440e-b6bd-8efb1ce7a539","Type":"ContainerStarted","Data":"20a83b8c3b0e3c957d627a12a5c57cf74f60897eed847c1741bc5f783652e25b"} Nov 01 22:31:22.629162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.629132 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:31:22.630689 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.630663 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" event={"ID":"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8","Type":"ContainerStarted","Data":"5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e"} Nov 01 22:31:22.630809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.630777 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:31:22.632245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.632221 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" event={"ID":"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d","Type":"ContainerStarted","Data":"ca097437a44483cb2e20571615b0d3112d8288d3c4b07a6de1da5146c80b1287"} Nov 01 22:31:22.632344 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.632257 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" event={"ID":"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d","Type":"ContainerStarted","Data":"0195f83f74c3fbd555b202daf5cdc5973170412445bfe5c413e09c123ac7fae7"} Nov 01 22:31:22.632443 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.632425 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:31:22.667123 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.667069 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" podStartSLOduration=2.667056858 podStartE2EDuration="2.667056858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:22.66621771 +0000 UTC m=+486.450656633" watchObservedRunningTime="2026-11-01 22:31:22.667056858 +0000 UTC m=+486.451495779" Nov 01 22:31:22.667290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.667172 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" podStartSLOduration=1.6671687450000001 podStartE2EDuration="1.667168745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:22.648801859 +0000 UTC m=+486.433240781" watchObservedRunningTime="2026-11-01 22:31:22.667168745 +0000 UTC m=+486.451607666" Nov 01 22:31:22.684018 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.683965 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" podStartSLOduration=1.683950534 podStartE2EDuration="1.683950534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:22.682209768 +0000 UTC m=+486.466648689" watchObservedRunningTime="2026-11-01 22:31:22.683950534 +0000 UTC m=+486.468389470" Nov 01 22:31:22.835005 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:22.834928 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:23.113283 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:23.113258 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l"] Nov 01 22:31:23.115791 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:31:23.115767 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cb80d1_5953_4b69_9c6b_51eef06e8336.slice/crio-39e2f14dce4fbab84d6d3c4b7e7db90fcfd2f85c43b5e92c4a2e88e8b71e7a3b WatchSource:0}: Error finding container 39e2f14dce4fbab84d6d3c4b7e7db90fcfd2f85c43b5e92c4a2e88e8b71e7a3b: Status 404 returned error can't find the container with id 39e2f14dce4fbab84d6d3c4b7e7db90fcfd2f85c43b5e92c4a2e88e8b71e7a3b Nov 01 22:31:23.639141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:23.639083 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" event={"ID":"73cb80d1-5953-4b69-9c6b-51eef06e8336","Type":"ContainerStarted","Data":"57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1"} Nov 01 22:31:23.639598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:23.639189 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" event={"ID":"73cb80d1-5953-4b69-9c6b-51eef06e8336","Type":"ContainerStarted","Data":"39e2f14dce4fbab84d6d3c4b7e7db90fcfd2f85c43b5e92c4a2e88e8b71e7a3b"} Nov 01 22:31:23.658118 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:23.658055 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" podStartSLOduration=2.21268646 podStartE2EDuration="2.658041117s" totalImagesPullingTime="445.354657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:23.655213677 +0000 UTC m=+487.439652603" watchObservedRunningTime="2026-11-01 22:31:23.658041117 +0000 UTC m=+487.442480037" Nov 01 22:31:24.644060 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:24.644033 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:25.178315 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.178281 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6"] Nov 01 22:31:25.183243 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.183221 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.192241 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.192216 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-b28a40-serving-cert\"" Nov 01 22:31:25.195276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.195252 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6"] Nov 01 22:31:25.206542 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.206513 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.206687 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.206663 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-metrics-certs\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.206743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.206697 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjk8\" (UniqueName: \"kubernetes.io/projected/ff764f93-2273-4694-bb85-72533fc7161c-kube-api-access-9qjk8\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.206743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.206728 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff764f93-2273-4694-bb85-72533fc7161c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.307481 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.307446 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-metrics-certs\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.307677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.307487 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjk8\" (UniqueName: \"kubernetes.io/projected/ff764f93-2273-4694-bb85-72533fc7161c-kube-api-access-9qjk8\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.307677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.307525 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff764f93-2273-4694-bb85-72533fc7161c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.307677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.307629 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.308273 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.308250 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff764f93-2273-4694-bb85-72533fc7161c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.310039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.310009 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.310039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.310032 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-metrics-certs\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.316703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.316672 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjk8\" (UniqueName: \"kubernetes.io/projected/ff764f93-2273-4694-bb85-72533fc7161c-kube-api-access-9qjk8\") pod \"maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.499228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.499095 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:25.773096 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:25.773071 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6"] Nov 01 22:31:25.775097 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:31:25.775068 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff764f93_2273_4694_bb85_72533fc7161c.slice/crio-9bb6878005ea36a56bf4e679ab2af7c64e60979aac7e350a4a1a70d76f209d1e WatchSource:0}: Error finding container 9bb6878005ea36a56bf4e679ab2af7c64e60979aac7e350a4a1a70d76f209d1e: Status 404 returned error can't find the container with id 9bb6878005ea36a56bf4e679ab2af7c64e60979aac7e350a4a1a70d76f209d1e Nov 01 22:31:26.627460 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:26.627385 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:31:26.660370 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:26.660318 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" event={"ID":"ff764f93-2273-4694-bb85-72533fc7161c","Type":"ContainerStarted","Data":"f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d"} Nov 01 22:31:26.660587 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:26.660561 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:26.660748 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:26.660616 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" event={"ID":"ff764f93-2273-4694-bb85-72533fc7161c","Type":"ContainerStarted","Data":"9bb6878005ea36a56bf4e679ab2af7c64e60979aac7e350a4a1a70d76f209d1e"} Nov 01 22:31:26.685636 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:26.685567 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" podStartSLOduration=1.28783744 podStartE2EDuration="1.685547549s" totalImagesPullingTime="397.710109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:26.683278369 +0000 UTC m=+490.467717291" watchObservedRunningTime="2026-11-01 22:31:26.685547549 +0000 UTC m=+490.469986475" Nov 01 22:31:27.662131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:27.661423 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:31:30.679132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:30.678340 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:31:32.719326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:32.719300 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:31:33.666712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:33.665980 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:31:33.666712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:33.666304 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:31:33.666712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:33.666389 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:31:38.971375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:38.816743 6466 generic.go:356] "Generic (PLEG): container finished" podID="53147cfc-27a6-4f61-863a-732bfd063831" containerID="ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98" exitCode=0 Nov 01 22:31:38.971375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:38.816826 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" event={"ID":"53147cfc-27a6-4f61-863a-732bfd063831","Type":"ContainerDied","Data":"ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98"} Nov 01 22:31:38.971375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:38.817437 6466 scope.go:122] "RemoveContainer" containerID="ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98" Nov 01 22:31:38.971375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:38.819237 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:38.971375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:38.819324 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:38.971375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:38.819368 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:39.824586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.824552 6466 generic.go:356] "Generic (PLEG): container finished" podID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerID="46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91" exitCode=0 Nov 01 22:31:39.824752 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.824621 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" event={"ID":"d52a7224-6cd7-44ea-9bf5-d16246da3250","Type":"ContainerDied","Data":"46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91"} Nov 01 22:31:39.825102 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.825083 6466 scope.go:122] "RemoveContainer" containerID="46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91" Nov 01 22:31:39.826583 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.826543 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:39.826666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.826629 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:39.826717 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.826664 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:39.826770 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.826718 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" event={"ID":"53147cfc-27a6-4f61-863a-732bfd063831","Type":"ContainerStarted","Data":"741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4"} Nov 01 22:31:39.828416 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.828395 6466 generic.go:356] "Generic (PLEG): container finished" podID="9a12f352-2870-4eb9-992f-d1d12348f299" containerID="bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8" exitCode=0 Nov 01 22:31:39.828507 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.828446 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" event={"ID":"9a12f352-2870-4eb9-992f-d1d12348f299","Type":"ContainerDied","Data":"bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8"} Nov 01 22:31:39.828762 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.828746 6466 scope.go:122] "RemoveContainer" containerID="bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8" Nov 01 22:31:39.829836 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.829810 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:39.829903 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.829888 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:39.829945 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:39.829927 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:40.371144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:40.368752 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:40.371144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:40.368875 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-whtxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 22:31:40.371144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:40.368920 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 22:31:40.854132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:40.852132 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" event={"ID":"d52a7224-6cd7-44ea-9bf5-d16246da3250","Type":"ContainerStarted","Data":"1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53"} Nov 01 22:31:40.892128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:40.891843 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" event={"ID":"9a12f352-2870-4eb9-992f-d1d12348f299","Type":"ContainerStarted","Data":"70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84"} Nov 01 22:31:40.912168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:40.903059 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:40.912168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:40.903201 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:31:40.912168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:40.903246 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:31:41.267141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:41.258587 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:41.267141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:41.258781 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 22:31:41.267141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:41.258829 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 22:31:41.370129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:41.369578 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-whtxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 22:31:41.370129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:41.369659 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 22:31:41.908131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:41.907291 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:31:41.908131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:41.907366 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:31:42.259201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:42.258298 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 22:31:42.259201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:42.258376 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 22:31:42.368133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:42.367057 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-whtxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 22:31:42.368133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:42.367152 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 22:31:42.902146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:42.901537 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:31:42.902146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:42.901632 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:31:43.274185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:43.273636 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 22:31:43.274185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:43.273696 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 22:31:43.370149 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:43.365935 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-whtxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 22:31:43.370149 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:43.365998 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 22:31:43.901067 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:43.900562 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:31:43.901067 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:43.900630 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:31:44.270184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:44.257389 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 22:31:44.270184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:44.257461 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 22:31:44.372314 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:44.372259 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-whtxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 22:31:44.372959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:44.372904 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 22:31:44.902977 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:44.902512 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:31:44.902977 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:44.902571 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:31:45.257214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.255896 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 22:31:45.257214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.255960 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 22:31:45.429978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.429190 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:45.430475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.430308 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:45.431467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.431393 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:31:45.900052 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.899363 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:31:45.900052 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.899436 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:31:45.995443 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.995409 6466 generic.go:356] "Generic (PLEG): container finished" podID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerID="3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593" exitCode=0 Nov 01 22:31:45.995617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.995481 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" event={"ID":"8831591d-4318-4b8d-b6bc-9790a73d8c67","Type":"ContainerDied","Data":"3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593"} Nov 01 22:31:45.996034 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.995952 6466 scope.go:122] "RemoveContainer" containerID="3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593" Nov 01 22:31:45.997139 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.997090 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:45.997215 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.997202 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:45.997250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:45.997239 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:31:46.266537 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:46.256982 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 22:31:46.266537 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:46.257041 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 22:31:46.589142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:46.589033 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:46.589142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:46.589130 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:46.589551 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:46.589155 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:47.006833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:47.006796 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" event={"ID":"8831591d-4318-4b8d-b6bc-9790a73d8c67","Type":"ContainerStarted","Data":"309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3"} Nov 01 22:31:47.180705 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:47.180648 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:47.182675 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:47.182650 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:31:47.289381 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:47.287861 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:47.289381 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:47.289041 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:31:47.407267 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:47.407242 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:47.408414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:47.408382 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 22:31:47.408559 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:47.408440 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 22:31:48.410433 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:48.410397 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 22:31:48.410893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:48.410465 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 22:31:49.293594 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:31:49.293561 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:31:49.407766 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:49.407736 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 22:31:49.408014 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:49.407973 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 22:31:50.407103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:50.407065 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 22:31:50.407529 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:50.407160 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 22:31:51.407458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:51.407425 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 22:31:51.407848 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:51.407480 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 22:31:52.494995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:52.494914 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:52.495559 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:52.495520 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:31:52.496770 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:31:52.496744 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:32:19.302842 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:32:19.302804 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:32:44.004782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.004733 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2"] Nov 01 22:32:44.009229 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.009196 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.012766 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.012743 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-aa441b-openshift-default-dockercfg-kjt95\"" Nov 01 22:32:44.018352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.018308 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2"] Nov 01 22:32:44.099662 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.099625 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16da46c6-bb20-4107-b575-a886d44abd6e-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.099662 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.099672 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.099882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.099701 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9xp8\" (UniqueName: \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-kube-api-access-v9xp8\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.099882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.099779 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.099882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.099829 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.099984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.099949 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.099984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.099969 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16da46c6-bb20-4107-b575-a886d44abd6e-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.100088 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.100068 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.100157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.100102 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.201548 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.201512 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.201548 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.201549 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16da46c6-bb20-4107-b575-a886d44abd6e-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.201792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.201610 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.201792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.201633 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.201792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.201677 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16da46c6-bb20-4107-b575-a886d44abd6e-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.201974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.201796 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.201974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.201838 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9xp8\" (UniqueName: \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-kube-api-access-v9xp8\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.201974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.201870 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.201974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.201920 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.202212 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.202042 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.202212 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.202142 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.202339 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.202318 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.202413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.202393 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.202470 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.202415 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16da46c6-bb20-4107-b575-a886d44abd6e-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.204044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.204024 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.204160 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.204139 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16da46c6-bb20-4107-b575-a886d44abd6e-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.210213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.210178 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9xp8\" (UniqueName: \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-kube-api-access-v9xp8\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.210213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.210192 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.325555 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.325468 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:44.606164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.606134 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2"] Nov 01 22:32:44.607836 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:32:44.607809 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16da46c6_bb20_4107_b575_a886d44abd6e.slice/crio-6daefffbd353ce9d3295f35d8e4da413a12a24debd8195df24bd6541558dd4bb WatchSource:0}: Error finding container 6daefffbd353ce9d3295f35d8e4da413a12a24debd8195df24bd6541558dd4bb: Status 404 returned error can't find the container with id 6daefffbd353ce9d3295f35d8e4da413a12a24debd8195df24bd6541558dd4bb Nov 01 22:32:44.610228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.610197 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:32:44.610306 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.610261 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:32:44.610306 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:44.610290 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:32:45.358691 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:45.358650 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" event={"ID":"16da46c6-bb20-4107-b575-a886d44abd6e","Type":"ContainerStarted","Data":"ed55ee8cc11f45ca456d5a6891d7bdc8551095232587c3878a24bedbc7046b36"} Nov 01 22:32:45.359202 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:45.358720 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" event={"ID":"16da46c6-bb20-4107-b575-a886d44abd6e","Type":"ContainerStarted","Data":"6daefffbd353ce9d3295f35d8e4da413a12a24debd8195df24bd6541558dd4bb"} Nov 01 22:32:45.394811 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:45.394745 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" podStartSLOduration=2.3947259819999998 podStartE2EDuration="2.394725982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:45.392638503 +0000 UTC m=+569.177077448" watchObservedRunningTime="2026-11-01 22:32:45.394725982 +0000 UTC m=+569.179164903" Nov 01 22:32:46.326205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:46.326162 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:46.327974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:46.327937 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:32:46.328135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:46.327984 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 22:32:47.326038 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:47.326003 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:32:47.326459 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:47.326057 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 22:32:48.325841 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:48.325803 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:32:48.326018 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:48.325866 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 22:32:49.289502 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:32:49.289463 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:32:49.326053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:49.326023 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:32:49.326257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:49.326072 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 22:32:50.326195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:50.326156 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:32:50.326687 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:50.326222 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 22:32:51.326699 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:51.326660 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 01 22:32:51.327247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:51.326727 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 01 22:32:52.330476 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:52.330446 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:52.330963 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:52.330861 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:52.331571 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:52.331551 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:32:58.818374 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.818340 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph"] Nov 01 22:32:58.822547 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.822524 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:58.832845 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.832821 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-00e49d-openshift-default-dockercfg-xhjdm\"" Nov 01 22:32:58.837689 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.837666 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph"] Nov 01 22:32:58.907427 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.907391 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:58.907612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.907553 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:58.907612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.907593 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:58.907694 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.907629 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:58.907751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.907735 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:58.907804 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.907793 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:58.907847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.907832 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:58.907897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.907885 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:58.908002 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:58.907983 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdqdz\" (UniqueName: \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-kube-api-access-kdqdz\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.009613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.009576 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.009819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.009657 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.009819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.009716 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.009819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.009788 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.010011 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.009820 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.010011 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.009844 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.010011 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.009894 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdqdz\" (UniqueName: \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-kube-api-access-kdqdz\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.010011 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.009944 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.010250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.010057 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.010250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.010097 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.010250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.010145 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.010662 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.010618 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.010662 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.010656 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.010876 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.010852 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.012713 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.012682 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.012853 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.012837 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.025718 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.025693 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdqdz\" (UniqueName: \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-kube-api-access-kdqdz\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.025908 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.025892 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.139243 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.139199 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:32:59.161719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.161686 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4"] Nov 01 22:32:59.167573 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.167547 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.178068 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.178037 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4"] Nov 01 22:32:59.218277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.209061 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e25182-openshift-default-dockercfg-pf4mc\"" Nov 01 22:32:59.218277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.213287 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.218277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.213440 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.218277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.213513 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04419d55-67d8-420d-8f48-f85969496c9d-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.218277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.214249 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbjd\" (UniqueName: \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-kube-api-access-4wbjd\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.218277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.214301 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.218277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.214352 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04419d55-67d8-420d-8f48-f85969496c9d-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.218277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.214489 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.218277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.214544 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.218277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.214584 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.315488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.315452 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.315667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.315504 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.315667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.315541 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.315667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.315570 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.315667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.315594 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.315871 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.315742 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04419d55-67d8-420d-8f48-f85969496c9d-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.315871 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.315852 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbjd\" (UniqueName: \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-kube-api-access-4wbjd\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.315971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.315898 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.315971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.315920 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.315971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.315935 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04419d55-67d8-420d-8f48-f85969496c9d-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.316149 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.316005 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.316149 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.316037 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.316385 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.316364 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.316515 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.316495 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04419d55-67d8-420d-8f48-f85969496c9d-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.318059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.318037 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.318187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.318165 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04419d55-67d8-420d-8f48-f85969496c9d-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.325489 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.325463 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbjd\" (UniqueName: \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-kube-api-access-4wbjd\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.325692 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.325672 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.454615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.454588 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph"] Nov 01 22:32:59.456615 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:32:59.456579 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4da916a_f9ac_4cd2_ad7d_4c6dd034f375.slice/crio-2fe4dfe4166b19fdac8928a57669921c07d801c5176606f17199cffd87981877 WatchSource:0}: Error finding container 2fe4dfe4166b19fdac8928a57669921c07d801c5176606f17199cffd87981877: Status 404 returned error can't find the container with id 2fe4dfe4166b19fdac8928a57669921c07d801c5176606f17199cffd87981877 Nov 01 22:32:59.459309 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.459271 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:32:59.459387 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.459359 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:32:59.459420 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.459407 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:32:59.493945 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.493917 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:32:59.784813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.784784 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4"] Nov 01 22:32:59.786341 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:32:59.786305 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04419d55_67d8_420d_8f48_f85969496c9d.slice/crio-df08251c3009f59005902395879bc0fd58bc281fd6ba9b13fa9acee627ba1936 WatchSource:0}: Error finding container df08251c3009f59005902395879bc0fd58bc281fd6ba9b13fa9acee627ba1936: Status 404 returned error can't find the container with id df08251c3009f59005902395879bc0fd58bc281fd6ba9b13fa9acee627ba1936 Nov 01 22:32:59.788917 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.788883 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:32:59.789013 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.788964 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:32:59.789013 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:32:59.789007 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:00.488156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:00.486838 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" event={"ID":"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375","Type":"ContainerStarted","Data":"f5f3dfd65610a547b18be189d9ec00596b10b66d104d5d810ed74db07acfa667"} Nov 01 22:33:00.488156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:00.486924 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" event={"ID":"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375","Type":"ContainerStarted","Data":"2fe4dfe4166b19fdac8928a57669921c07d801c5176606f17199cffd87981877"} Nov 01 22:33:00.490183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:00.490153 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" event={"ID":"04419d55-67d8-420d-8f48-f85969496c9d","Type":"ContainerStarted","Data":"0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50"} Nov 01 22:33:00.490317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:00.490203 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" event={"ID":"04419d55-67d8-420d-8f48-f85969496c9d","Type":"ContainerStarted","Data":"df08251c3009f59005902395879bc0fd58bc281fd6ba9b13fa9acee627ba1936"} Nov 01 22:33:00.494913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:00.494892 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:33:00.496091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:00.496072 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 22:33:00.496183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:00.496130 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" podUID="04419d55-67d8-420d-8f48-f85969496c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 22:33:00.514984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:00.514930 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" podStartSLOduration=2.514912792 podStartE2EDuration="2.514912792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:00.513782836 +0000 UTC m=+584.298221757" watchObservedRunningTime="2026-11-01 22:33:00.514912792 +0000 UTC m=+584.299351713" Nov 01 22:33:00.585186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:00.565704 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" podStartSLOduration=1.5656792959999999 podStartE2EDuration="1.565679296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:00.556390372 +0000 UTC m=+584.340829334" watchObservedRunningTime="2026-11-01 22:33:00.565679296 +0000 UTC m=+584.350118218" Nov 01 22:33:01.154132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:01.150121 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:33:01.154132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:01.153438 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 22:33:01.154132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:01.153498 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" podUID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 22:33:01.495414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:01.495314 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 22:33:01.495414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:01.495396 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" podUID="04419d55-67d8-420d-8f48-f85969496c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 22:33:02.147135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:02.143589 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 22:33:02.147135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:02.143655 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" podUID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 22:33:02.494917 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:02.494826 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 22:33:02.494917 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:02.494879 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" podUID="04419d55-67d8-420d-8f48-f85969496c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 22:33:03.143686 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:03.143639 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 22:33:03.144823 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:03.144296 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" podUID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 22:33:03.494390 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:03.494298 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 22:33:03.494390 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:03.494357 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" podUID="04419d55-67d8-420d-8f48-f85969496c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 22:33:04.144132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:04.143562 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 22:33:04.144132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:04.143628 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" podUID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 22:33:04.495373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:04.495279 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 22:33:04.495373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:04.495342 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" podUID="04419d55-67d8-420d-8f48-f85969496c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 22:33:05.151134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:05.143653 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 01 22:33:05.151134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:05.143738 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" podUID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 01 22:33:05.494834 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:05.494750 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 01 22:33:05.494834 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:05.494805 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" podUID="04419d55-67d8-420d-8f48-f85969496c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 01 22:33:06.344422 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:06.344379 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:33:06.346165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:06.346125 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:33:06.346921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:06.346900 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:33:06.745933 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:06.745896 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:33:06.746828 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:06.746800 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:33:06.749172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:06.749146 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:33:19.280238 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:33:19.280198 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:33:44.253163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.253126 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst"] Nov 01 22:33:44.257854 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.257827 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.290090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.290050 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst"] Nov 01 22:33:44.347410 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.347377 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.347612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.347425 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.347612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.347452 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6496\" (UniqueName: \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-kube-api-access-q6496\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.347612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.347529 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a544a3-27d3-43ff-98c4-b47975cfc225-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.347612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.347603 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.347771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.347693 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.347771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.347751 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.347833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.347798 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.347866 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.347852 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.448948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.448909 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.448948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.448956 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449217 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.448999 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449217 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.449017 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449217 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.449050 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449217 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.449182 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449217 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.449212 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.449240 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6496\" (UniqueName: \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-kube-api-access-q6496\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.449269 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a544a3-27d3-43ff-98c4-b47975cfc225-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.449421 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.449441 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449686 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.449566 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449686 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.449669 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.449999 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.449980 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a544a3-27d3-43ff-98c4-b47975cfc225-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.451536 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.451511 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.451749 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.451731 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.458402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.458374 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.458503 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.458467 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6496\" (UniqueName: \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-kube-api-access-q6496\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.602405 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.602308 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:44.888982 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.886215 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst"] Nov 01 22:33:44.892440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.891896 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:44.892440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.892050 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:44.892440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.892158 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:44.935759 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:44.935141 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" event={"ID":"42a544a3-27d3-43ff-98c4-b47975cfc225","Type":"ContainerStarted","Data":"22ae6dc91cc8dd6c417eeb09757646bbd516d43cb7336cccf89481bcf8c01c2c"} Nov 01 22:33:45.965615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:45.965572 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" event={"ID":"42a544a3-27d3-43ff-98c4-b47975cfc225","Type":"ContainerStarted","Data":"8d6985449d07ccb5ed805713cbcbb55ae07e7d56c2ab47af9dc8467cc30207e3"} Nov 01 22:33:45.988912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:45.988860 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" podStartSLOduration=1.988844856 podStartE2EDuration="1.988844856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:45.987278355 +0000 UTC m=+629.771717278" watchObservedRunningTime="2026-11-01 22:33:45.988844856 +0000 UTC m=+629.773283777" Nov 01 22:33:46.602912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:46.602881 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:46.604573 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:46.604547 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 22:33:46.604716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:46.604599 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 22:33:47.603307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:47.603270 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 22:33:47.603747 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:47.603325 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 22:33:48.613132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:48.603564 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 22:33:48.613132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:48.603634 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 22:33:49.597304 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:33:49.597257 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:33:49.604539 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:49.604502 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 22:33:49.604886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:49.604851 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 22:33:50.602826 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:50.602795 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 22:33:50.603225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:50.602853 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 22:33:51.603789 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:51.603752 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 22:33:51.604214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:51.603811 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 22:33:52.813438 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.813399 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6"] Nov 01 22:33:52.818568 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.818542 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.831241 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.831200 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc"] Nov 01 22:33:52.837048 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.837019 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6"] Nov 01 22:33:52.837233 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.837193 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.853623 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.853459 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc"] Nov 01 22:33:52.860980 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.860945 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2"] Nov 01 22:33:52.867093 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.866304 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:52.889340 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.889301 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.889636 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.889612 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.889791 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.889773 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.889985 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.889948 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.890081 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.889997 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.890081 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890029 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.890081 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890062 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnw92\" (UniqueName: \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-kube-api-access-gnw92\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.890290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890142 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.890290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890205 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.890290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890243 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fs2b\" (UniqueName: \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-kube-api-access-9fs2b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.890290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890271 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45719f2c-6797-450c-8089-e90c493e3b3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.890569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890307 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.890569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890373 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.890569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890397 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.890569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890438 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45719f2c-6797-450c-8089-e90c493e3b3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.890569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890463 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.890569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890516 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.890569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.890557 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.892009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.891983 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2"] Nov 01 22:33:52.992227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992188 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.992411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992275 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.992411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992309 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:52.992411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992360 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.992411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992396 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.992656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992428 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:52.992656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992458 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d0e2c6-68dd-486b-a461-1564218a792b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:52.992656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992495 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:52.992656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992524 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:52.992656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992571 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.992656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992600 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.992952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992667 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.992952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992936 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.993059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.992986 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.993157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993081 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.993290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993262 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnw92\" (UniqueName: \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-kube-api-access-gnw92\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.993384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993365 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.993448 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993363 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.993507 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993469 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.993562 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993507 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqtn6\" (UniqueName: \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-kube-api-access-sqtn6\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:52.993562 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993542 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fs2b\" (UniqueName: \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-kube-api-access-9fs2b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.993668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993571 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45719f2c-6797-450c-8089-e90c493e3b3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.993668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993610 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.993668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993659 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:52.993789 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993672 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.993844 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.993826 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.994135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.994077 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.994251 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.994208 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:52.994312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.994260 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.994312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.994289 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.994418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.994336 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45719f2c-6797-450c-8089-e90c493e3b3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.994418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.994359 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.994528 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.994444 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:52.994528 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.994477 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.995096 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.994806 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45719f2c-6797-450c-8089-e90c493e3b3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.995096 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.994837 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.995449 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.995428 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.996893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.996866 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.997883 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.997667 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:52.998688 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.998663 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45719f2c-6797-450c-8089-e90c493e3b3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:52.998879 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:52.998858 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:53.001613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.001585 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:53.004838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.004809 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnw92\" (UniqueName: \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-kube-api-access-gnw92\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dgmc6\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:53.004838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.004831 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fs2b\" (UniqueName: \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-kube-api-access-9fs2b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:53.005969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.005949 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ggzxc\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:53.095356 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.095253 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqtn6\" (UniqueName: \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-kube-api-access-sqtn6\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.095356 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.095314 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.095356 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.095350 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.095645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.095414 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.095645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.095481 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.095645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.095525 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.095645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.095551 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d0e2c6-68dd-486b-a461-1564218a792b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.095645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.095590 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.095645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.095617 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.095952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.095803 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.095952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.095914 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.096159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.096093 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.096940 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.096850 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d0e2c6-68dd-486b-a461-1564218a792b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.097059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.097025 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.098623 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.098593 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.098839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.098817 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.104255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.104234 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqtn6\" (UniqueName: \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-kube-api-access-sqtn6\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.104348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.104318 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q5hw2\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.138764 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.138718 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:53.159660 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.159601 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:53.185676 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.185318 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:53.444872 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.444829 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6"] Nov 01 22:33:53.450933 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:33:53.450899 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45719f2c_6797_450c_8089_e90c493e3b3a.slice/crio-0b1fe3a4eb740f9ba8a2efeccdeb12ff5a80470fc8a0be2378850e2f4b44f96e WatchSource:0}: Error finding container 0b1fe3a4eb740f9ba8a2efeccdeb12ff5a80470fc8a0be2378850e2f4b44f96e: Status 404 returned error can't find the container with id 0b1fe3a4eb740f9ba8a2efeccdeb12ff5a80470fc8a0be2378850e2f4b44f96e Nov 01 22:33:53.453714 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.453671 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:53.453818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.453767 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:53.453883 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.453818 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:53.485276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.485246 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:53.487286 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.485982 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:53.488469 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.488449 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:33:53.501057 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.499852 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc"] Nov 01 22:33:53.512329 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:33:53.512303 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aeac0ad_04ea_4c1b_b273_a0d708cc552f.slice/crio-c69a3a791e36200ec70051964c81880ad4654f98d7a728a98fa1810d5cab8cb5 WatchSource:0}: Error finding container c69a3a791e36200ec70051964c81880ad4654f98d7a728a98fa1810d5cab8cb5: Status 404 returned error can't find the container with id c69a3a791e36200ec70051964c81880ad4654f98d7a728a98fa1810d5cab8cb5 Nov 01 22:33:53.515284 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.515251 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:53.515390 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.515337 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:53.515390 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.515383 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:53.524726 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.524633 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2"] Nov 01 22:33:53.528799 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:33:53.528772 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d0e2c6_68dd_486b_a461_1564218a792b.slice/crio-b6512e815daae1b46c48e8795c54098787c65569bde5533e6872463312d5020b WatchSource:0}: Error finding container b6512e815daae1b46c48e8795c54098787c65569bde5533e6872463312d5020b: Status 404 returned error can't find the container with id b6512e815daae1b46c48e8795c54098787c65569bde5533e6872463312d5020b Nov 01 22:33:53.533189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.533145 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:53.533313 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.533276 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:53.533374 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:53.533324 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:33:54.100666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.100627 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" event={"ID":"d9d0e2c6-68dd-486b-a461-1564218a792b","Type":"ContainerStarted","Data":"9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8"} Nov 01 22:33:54.101485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.101456 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" event={"ID":"d9d0e2c6-68dd-486b-a461-1564218a792b","Type":"ContainerStarted","Data":"b6512e815daae1b46c48e8795c54098787c65569bde5533e6872463312d5020b"} Nov 01 22:33:54.113931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.113886 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" event={"ID":"45719f2c-6797-450c-8089-e90c493e3b3a","Type":"ContainerStarted","Data":"886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f"} Nov 01 22:33:54.114258 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.114219 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" event={"ID":"45719f2c-6797-450c-8089-e90c493e3b3a","Type":"ContainerStarted","Data":"0b1fe3a4eb740f9ba8a2efeccdeb12ff5a80470fc8a0be2378850e2f4b44f96e"} Nov 01 22:33:54.119693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.119591 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" event={"ID":"7aeac0ad-04ea-4c1b-b273-a0d708cc552f","Type":"ContainerStarted","Data":"3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5"} Nov 01 22:33:54.119693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.119637 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" event={"ID":"7aeac0ad-04ea-4c1b-b273-a0d708cc552f","Type":"ContainerStarted","Data":"c69a3a791e36200ec70051964c81880ad4654f98d7a728a98fa1810d5cab8cb5"} Nov 01 22:33:54.139715 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.139686 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:33:54.141289 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.141257 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 01 22:33:54.141424 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.141308 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 01 22:33:54.153321 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.153260 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podStartSLOduration=2.153239051 podStartE2EDuration="2.153239051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:54.128655825 +0000 UTC m=+637.913094747" watchObservedRunningTime="2026-11-01 22:33:54.153239051 +0000 UTC m=+637.937677965" Nov 01 22:33:54.154084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.154047 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podStartSLOduration=2.1540347029999998 podStartE2EDuration="2.154034703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:54.151672349 +0000 UTC m=+637.936111283" watchObservedRunningTime="2026-11-01 22:33:54.154034703 +0000 UTC m=+637.938473629" Nov 01 22:33:54.161388 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.161322 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:33:54.162793 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.162763 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 22:33:54.162943 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.162818 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 22:33:54.173146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.170049 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podStartSLOduration=2.170030531 podStartE2EDuration="2.170030531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:54.169502419 +0000 UTC m=+637.953941345" watchObservedRunningTime="2026-11-01 22:33:54.170030531 +0000 UTC m=+637.954469452" Nov 01 22:33:54.186197 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.186167 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:33:54.187463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.187442 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 22:33:54.187520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:54.187489 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 22:33:55.248133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:55.247266 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 22:33:55.248133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:55.247341 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 22:33:55.248133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:55.247676 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 22:33:55.248133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:55.247708 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 22:33:55.248133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:55.247900 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 01 22:33:55.248133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:55.247926 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 01 22:33:56.140554 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:56.140459 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 01 22:33:56.140829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:56.140797 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 01 22:33:56.179135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:56.178713 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 22:33:56.179135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:56.178774 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 22:33:56.211133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:56.200708 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 22:33:56.211133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:56.200764 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 22:33:57.155173 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:57.142812 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 01 22:33:57.155173 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:57.142878 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 01 22:33:57.163131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:57.160632 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 22:33:57.163131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:57.160703 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 22:33:57.187129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:57.187024 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 22:33:57.187129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:57.187086 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 22:33:58.143137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:58.140639 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 01 22:33:58.143137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:58.140717 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 01 22:33:58.169172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:58.168228 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 22:33:58.169172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:58.168297 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 22:33:58.192135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:58.190564 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 22:33:58.192135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:58.190626 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 22:33:58.940134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:58.933662 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp"] Nov 01 22:33:59.079654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.078350 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp"] Nov 01 22:33:59.079654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.079288 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.109146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.102084 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.109146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.102189 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.109146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.102225 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.109146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.102270 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6qrw\" (UniqueName: \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-kube-api-access-c6qrw\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.109146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.102373 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.109146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.102414 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.109146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.102460 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.109146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.102490 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.109146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.102519 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.134543 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.131706 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn"] Nov 01 22:33:59.143130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.142998 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 01 22:33:59.143130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.143055 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 01 22:33:59.167458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.160198 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 22:33:59.167458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.160257 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 22:33:59.177168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.175405 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208300 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208343 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208376 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208406 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208450 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6qrw\" (UniqueName: \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-kube-api-access-c6qrw\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208517 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208587 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208628 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208652 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208711 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208752 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208774 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208804 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208856 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208920 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e843f55-141b-4fab-a772-fa84acc14d2e-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.208978 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv9xt\" (UniqueName: \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-kube-api-access-rv9xt\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.209011 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.209192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.209047 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.246512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.227845 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn"] Nov 01 22:33:59.246512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.229614 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 22:33:59.246512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.229756 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 22:33:59.246512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.234191 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n"] Nov 01 22:33:59.262608 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.262572 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.263136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.263091 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.264028 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.249813 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.265222 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.265203 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.265869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.265832 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.340260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.336431 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.389364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.365314 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.416948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.416916 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.419073 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.419045 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e843f55-141b-4fab-a772-fa84acc14d2e-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.419339 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.419319 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv9xt\" (UniqueName: \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-kube-api-access-rv9xt\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.419511 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.419494 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.419671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.419652 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.419791 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.419778 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.419968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.419954 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.420200 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.420178 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.421219 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.416997 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.422096 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.422073 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e843f55-141b-4fab-a772-fa84acc14d2e-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.422514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.422494 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.422960 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.422920 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.433884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.424001 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.433884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.432234 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.433884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.432449 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.433884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.433608 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.433884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.433630 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.434327 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.433884 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.434939 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.434913 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.445974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.445939 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6qrw\" (UniqueName: \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-kube-api-access-c6qrw\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.454093 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.453669 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv9xt\" (UniqueName: \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-kube-api-access-rv9xt\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-ktkbn\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.485468 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.484873 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n"] Nov 01 22:33:59.523090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.522499 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:33:59.524593 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.524377 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.524593 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.524447 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.524593 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.524473 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlnhh\" (UniqueName: \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-kube-api-access-hlnhh\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.524593 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.524538 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.524593 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.524563 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.525991 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.525958 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.526135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.526062 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.526210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.526185 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.526266 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.526226 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.628097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.628067 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.628307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.628133 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.628307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.628210 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.628307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.628267 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.628307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.628291 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlnhh\" (UniqueName: \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-kube-api-access-hlnhh\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.628535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.628353 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.628535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.628377 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.628535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.628404 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.628535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.628425 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.628535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.628514 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.628831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.628813 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.629352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.629330 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.629578 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.629557 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.629744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.629717 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.631290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.631267 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.631575 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.631555 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.680132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.679794 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.708072 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.708033 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlnhh\" (UniqueName: \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-kube-api-access-hlnhh\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:33:59.708606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.708501 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:33:59.827327 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:33:59.826951 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:34:00.105224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.046851 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp"] Nov 01 22:34:00.207573 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.206303 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:00.207573 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.206400 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:00.207573 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.206444 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:00.265411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.216327 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 01 22:34:00.265411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.216384 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 01 22:34:00.265411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.217856 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 22:34:00.265411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.217915 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 22:34:00.265411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.218254 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 22:34:00.265411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.218295 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 22:34:00.364289 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.364257 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn"] Nov 01 22:34:00.370243 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:34:00.368029 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e843f55_141b_4fab_a772_fa84acc14d2e.slice/crio-9b67a6b3f021e1549c6377de16f2daae1866629fee9773e79411decadaa124ab WatchSource:0}: Error finding container 9b67a6b3f021e1549c6377de16f2daae1866629fee9773e79411decadaa124ab: Status 404 returned error can't find the container with id 9b67a6b3f021e1549c6377de16f2daae1866629fee9773e79411decadaa124ab Nov 01 22:34:00.372629 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.372103 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:00.372629 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.372234 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:00.372629 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.372293 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:00.508480 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.508427 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n"] Nov 01 22:34:00.525484 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.524984 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:00.525484 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.525077 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:00.525484 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.525134 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:00.646812 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.646434 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" event={"ID":"b00c4715-e3be-4ea0-8ba6-6d78396fdf18","Type":"ContainerStarted","Data":"daa17676493be99ce254c5b267b265e52ead8462988c684230b640490acf1a99"} Nov 01 22:34:00.649025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.648989 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" event={"ID":"3e843f55-141b-4fab-a772-fa84acc14d2e","Type":"ContainerStarted","Data":"c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443"} Nov 01 22:34:00.649203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.649046 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" event={"ID":"3e843f55-141b-4fab-a772-fa84acc14d2e","Type":"ContainerStarted","Data":"9b67a6b3f021e1549c6377de16f2daae1866629fee9773e79411decadaa124ab"} Nov 01 22:34:00.654524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.651663 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" event={"ID":"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e","Type":"ContainerStarted","Data":"e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc"} Nov 01 22:34:00.654524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.651717 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" event={"ID":"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e","Type":"ContainerStarted","Data":"89d99d7b4e32c83ef3473bca0a44de9651c0c39d2aebc7eba76bd0f40f72d1ac"} Nov 01 22:34:00.674277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.674193 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podStartSLOduration=2.6741750829999997 podStartE2EDuration="2.674175083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:00.673409943 +0000 UTC m=+644.457848864" watchObservedRunningTime="2026-11-01 22:34:00.674175083 +0000 UTC m=+644.458614016" Nov 01 22:34:00.709262 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:00.709198 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:34:01.141195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.139643 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 01 22:34:01.141195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.139700 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 01 22:34:01.167140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.165497 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 22:34:01.167140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.165561 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 22:34:01.190143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.186495 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 22:34:01.190143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.186550 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 22:34:01.528221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.527188 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:34:01.528221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.527350 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 22:34:01.528221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.527403 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 22:34:01.712132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.711343 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 22:34:01.712132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.711408 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 22:34:01.735326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.731797 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" event={"ID":"b00c4715-e3be-4ea0-8ba6-6d78396fdf18","Type":"ContainerStarted","Data":"5cfe1758824da5e20ec321751dca2e6669576c740560ac37b024e511267b968b"} Nov 01 22:34:01.792785 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.792085 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" podStartSLOduration=3.792063572 podStartE2EDuration="3.792063572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:00.709815242 +0000 UTC m=+644.494254163" watchObservedRunningTime="2026-11-01 22:34:01.792063572 +0000 UTC m=+645.576502494" Nov 01 22:34:01.834392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.831634 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:34:01.834392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.831795 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 22:34:01.834392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:01.831840 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 22:34:02.158133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.142649 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 01 22:34:02.158133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.142741 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 01 22:34:02.184132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.173593 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 01 22:34:02.184132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.173655 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 01 22:34:02.188132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.186035 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 22:34:02.188132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.186098 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 22:34:02.535188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.527655 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 22:34:02.535188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.527739 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 22:34:02.716723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.716662 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 22:34:02.734697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.734624 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 22:34:02.828173 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.828068 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 22:34:02.828932 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:02.828392 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 22:34:03.540738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:03.540662 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 22:34:03.541400 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:03.541363 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 22:34:03.882196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:03.833573 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 22:34:03.882196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:03.833728 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 22:34:03.882196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:03.855249 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 22:34:03.882196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:03.855313 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 22:34:04.221134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.217237 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:04.221134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.217305 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:04.269139 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.268226 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:04.269139 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.268301 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:04.288652 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.278145 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:04.317255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.295229 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:04.563200 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.559193 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 22:34:04.563200 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.559252 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 22:34:04.716142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.715601 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 22:34:04.716142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.715683 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 22:34:04.834751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.834654 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 22:34:04.835035 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:04.834987 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 22:34:05.248607 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.236026 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:05.248607 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.236090 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:05.339154 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.339081 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:05.364159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.339172 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:05.398152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.397250 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:05.398152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.397314 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:05.562682 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.562590 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:34:05.564412 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.564392 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:34:05.604409 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.604376 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 22:34:05.632139 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.605946 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 22:34:05.632433 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.632398 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 22:34:05.641140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.628719 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:34:05.642566 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.642538 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:34:05.675333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.675303 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 22:34:05.709026 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.708995 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 22:34:05.709301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.709274 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 22:34:05.747305 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.747226 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" podStartSLOduration=6.747202199 podStartE2EDuration="6.747202199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:01.794338174 +0000 UTC m=+645.578777120" watchObservedRunningTime="2026-11-01 22:34:05.747202199 +0000 UTC m=+649.531641112" Nov 01 22:34:05.750934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.750651 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:34:05.760766 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.760734 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:34:05.815189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.810898 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 22:34:05.850170 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.832769 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 22:34:05.850170 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:05.832828 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 22:34:06.528129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:06.524353 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 22:34:06.528129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:06.524402 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 22:34:06.716779 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:06.716737 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 22:34:06.717654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:06.717528 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 22:34:06.828630 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:06.828520 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 22:34:06.828630 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:06.828581 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 22:34:07.524157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:07.523854 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 22:34:07.524157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:07.523930 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 22:34:07.713282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:07.713238 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 22:34:07.713550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:07.713504 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 22:34:07.827936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:07.827844 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 22:34:07.828343 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:07.827928 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 22:34:08.523986 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:08.523952 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 22:34:08.524195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:08.524018 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 22:34:08.709580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:08.709533 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 22:34:08.709799 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:08.709590 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 22:34:08.828575 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:08.828497 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 01 22:34:08.828941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:08.828569 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 01 22:34:09.778181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:09.778143 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:34:09.778575 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:09.778542 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:34:09.779509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:09.779477 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:34:09.822236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:09.822206 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:34:09.823246 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:09.823216 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:34:09.826700 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:09.826622 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:34:10.346379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:10.346350 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:34:10.346872 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:10.346804 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:34:10.347749 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:10.347733 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:34:14.230226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.230195 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr"] Nov 01 22:34:14.235569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.235545 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.246168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.246140 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr"] Nov 01 22:34:14.332617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.332582 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.332851 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.332633 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.332851 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.332752 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5c15c5-4f15-43d2-b26c-2ea276334184-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.332977 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.332949 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.333032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.332997 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.333082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.333070 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.333149 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.333128 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.333149 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.333145 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfdj\" (UniqueName: \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-kube-api-access-wkfdj\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.333260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.333193 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.434454 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.434417 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.434454 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.434458 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.434671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.434565 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.434671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.434606 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.434671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.434621 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfdj\" (UniqueName: \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-kube-api-access-wkfdj\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.434810 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.434689 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.434810 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.434751 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.434810 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.434799 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.434969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.434852 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5c15c5-4f15-43d2-b26c-2ea276334184-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.434969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.434948 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.435074 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.435001 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.435350 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.435324 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.435457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.435329 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.435738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.435720 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5c15c5-4f15-43d2-b26c-2ea276334184-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.436887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.436860 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.437348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.437328 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.442432 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.442412 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.442509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.442463 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfdj\" (UniqueName: \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-kube-api-access-wkfdj\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.559525 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.559442 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:14.842466 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.842436 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr"] Nov 01 22:34:14.843625 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:34:14.843593 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5c15c5_4f15_43d2_b26c_2ea276334184.slice/crio-c6d58eee70ac2019405040c5a4ebb53113e1ec852236df44703f1affc52435e4 WatchSource:0}: Error finding container c6d58eee70ac2019405040c5a4ebb53113e1ec852236df44703f1affc52435e4: Status 404 returned error can't find the container with id c6d58eee70ac2019405040c5a4ebb53113e1ec852236df44703f1affc52435e4 Nov 01 22:34:14.846219 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.846187 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:14.846323 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.846268 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:14.846323 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:14.846313 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:15.576743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.576708 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs"] Nov 01 22:34:15.580939 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.580916 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.585384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.585356 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-00e49d\"/\"kube-root-ca.crt\"" Nov 01 22:34:15.587032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.586881 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-00e49d\"/\"openshift-service-ca.crt\"" Nov 01 22:34:15.587032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.586906 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-00e49d\"/\"default-dockercfg-6hshw\"" Nov 01 22:34:15.587032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.587006 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-00e49d\"/\"distinct-w3-00e49d-kserve-self-signed-certs\"" Nov 01 22:34:15.589360 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.589339 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs"] Nov 01 22:34:15.676678 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.676642 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" event={"ID":"3b5c15c5-4f15-43d2-b26c-2ea276334184","Type":"ContainerStarted","Data":"aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c"} Nov 01 22:34:15.676869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.676712 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" event={"ID":"3b5c15c5-4f15-43d2-b26c-2ea276334184","Type":"ContainerStarted","Data":"c6d58eee70ac2019405040c5a4ebb53113e1ec852236df44703f1affc52435e4"} Nov 01 22:34:15.756595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.756548 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65v9c\" (UniqueName: \"kubernetes.io/projected/5c7c32d2-89e8-4703-9775-0a7a168b318e-kube-api-access-65v9c\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.756975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.756948 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7c32d2-89e8-4703-9775-0a7a168b318e-tls-certs\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.757126 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.757094 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-dshm\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.757405 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.757387 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-home\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.757555 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.757540 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-model-cache\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.757718 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.757704 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-tmp-dir\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.860030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.859937 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7c32d2-89e8-4703-9775-0a7a168b318e-tls-certs\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.861295 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.861268 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-dshm\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.861564 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.861547 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-home\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.861721 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.861705 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-model-cache\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.861909 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.861894 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-tmp-dir\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.862100 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.862080 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65v9c\" (UniqueName: \"kubernetes.io/projected/5c7c32d2-89e8-4703-9775-0a7a168b318e-kube-api-access-65v9c\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.881383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.863625 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-model-cache\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.882758 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.882062 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-home\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.882937 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.882319 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-tmp-dir\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.892921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.892886 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7c32d2-89e8-4703-9775-0a7a168b318e-tls-certs\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.900054 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.894433 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-dshm\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.900054 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.899989 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65v9c\" (UniqueName: \"kubernetes.io/projected/5c7c32d2-89e8-4703-9775-0a7a168b318e-kube-api-access-65v9c\") pod \"distinct-w3-00e49d-kserve-6485b75c54-h8lxs\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:15.900598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:15.900576 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:16.223875 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:16.223824 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" podStartSLOduration=2.2238100259999998 podStartE2EDuration="2.223810026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:15.736526522 +0000 UTC m=+659.520965469" watchObservedRunningTime="2026-11-01 22:34:16.223810026 +0000 UTC m=+660.008248947" Nov 01 22:34:16.226336 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:16.226314 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs"] Nov 01 22:34:16.229398 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:34:16.229370 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c7c32d2_89e8_4703_9775_0a7a168b318e.slice/crio-96572498bb62084fe89c6e8d58e4376552a7e6f91da9c44554ebb612d84c66ce WatchSource:0}: Error finding container 96572498bb62084fe89c6e8d58e4376552a7e6f91da9c44554ebb612d84c66ce: Status 404 returned error can't find the container with id 96572498bb62084fe89c6e8d58e4376552a7e6f91da9c44554ebb612d84c66ce Nov 01 22:34:16.559979 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:16.559897 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:16.583477 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:16.583436 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:34:16.584013 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:16.583502 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 22:34:16.684230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:16.684189 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" event={"ID":"5c7c32d2-89e8-4703-9775-0a7a168b318e","Type":"ContainerStarted","Data":"9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00"} Nov 01 22:34:16.684230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:16.684237 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" event={"ID":"5c7c32d2-89e8-4703-9775-0a7a168b318e","Type":"ContainerStarted","Data":"96572498bb62084fe89c6e8d58e4376552a7e6f91da9c44554ebb612d84c66ce"} Nov 01 22:34:16.701568 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:16.701514 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podStartSLOduration=1.701499687 podStartE2EDuration="1.701499687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:16.700333037 +0000 UTC m=+660.484771959" watchObservedRunningTime="2026-11-01 22:34:16.701499687 +0000 UTC m=+660.485938609" Nov 01 22:34:16.907133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:16.903214 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:17.560635 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:17.560591 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:34:17.560826 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:17.560663 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 22:34:18.560555 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:18.560515 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:34:18.560974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:18.560575 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 22:34:19.294854 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:34:19.294813 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:34:19.560569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:19.560476 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:34:19.561211 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:19.561159 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 22:34:20.560616 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:20.560574 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:34:20.562796 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:20.562709 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 22:34:21.560073 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:21.560029 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:34:21.560247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:21.560176 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 22:34:22.564359 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:22.564329 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:22.564790 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:22.564736 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:22.565748 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:22.565726 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:34:25.901040 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:25.901001 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:25.913708 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:25.913677 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:26.773929 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:26.773900 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:34:33.447241 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:33.443208 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:33.447241 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:33.443289 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:43.890574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.890539 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk"] Nov 01 22:34:43.895760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.895742 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:43.909382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.909353 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk"] Nov 01 22:34:43.970889 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.970853 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:43.971097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.970941 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:43.971215 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.971080 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:43.971215 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.971159 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk8rt\" (UniqueName: \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-kube-api-access-vk8rt\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:43.971215 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.971200 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:43.971374 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.971226 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:43.971427 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.971374 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:43.971482 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.971425 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:43.971530 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:43.971496 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.074747 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.074702 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.075095 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.075067 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.075481 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.075453 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.075673 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.075649 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.075849 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.075690 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.075918 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.075844 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.075918 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.075895 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.076012 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.075955 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.076012 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.075994 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk8rt\" (UniqueName: \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-kube-api-access-vk8rt\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.076806 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.076768 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.079641 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.078190 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.080009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.079942 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.089777 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.089743 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr"] Nov 01 22:34:44.093138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.092389 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.104147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.097314 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk8rt\" (UniqueName: \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-kube-api-access-vk8rt\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.111039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.110959 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.113901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.113877 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr"] Nov 01 22:34:44.114213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.114198 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.134723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.119872 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.135042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.127492 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv"] Nov 01 22:34:44.135402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.135376 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.135625 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.132717 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.143918 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.143853 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.144162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.143899 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv"] Nov 01 22:34:44.177008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.176980 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.177354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177339 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.177543 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177506 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.177614 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177589 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.177658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177626 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1daa908-9812-4ecf-923d-160fca8742c7-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.177658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177650 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.177729 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177678 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.177729 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177696 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.177729 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177712 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.177878 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177736 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.177878 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177765 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.177878 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177798 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.177878 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177829 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.177878 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177845 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cj2g\" (UniqueName: \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-kube-api-access-8cj2g\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.177878 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177874 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.178219 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177910 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1daa908-9812-4ecf-923d-160fca8742c7-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.178219 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177930 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr42f\" (UniqueName: \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-kube-api-access-mr42f\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.178219 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.177979 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.211006 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.210972 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:44.259039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.259008 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb"] Nov 01 22:34:44.269171 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.266988 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293173 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293282 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293325 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293362 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293408 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293455 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1daa908-9812-4ecf-923d-160fca8742c7-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293490 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293537 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293566 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293591 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293626 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293671 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293724 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293774 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293797 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cj2g\" (UniqueName: \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-kube-api-access-8cj2g\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293844 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293891 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1daa908-9812-4ecf-923d-160fca8742c7-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.293923 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr42f\" (UniqueName: \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-kube-api-access-mr42f\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.295288 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.295809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.295766 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.297633 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.295840 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.297633 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.295959 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.297633 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.296034 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.297633 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.297319 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.301392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.298463 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.301392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.299035 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.301392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.299625 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1daa908-9812-4ecf-923d-160fca8742c7-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.301392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.300864 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.301392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.301274 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.305985 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.305953 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.307387 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.307345 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cj2g\" (UniqueName: \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-kube-api-access-8cj2g\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.309545 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.309093 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb"] Nov 01 22:34:44.312891 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.312838 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.316652 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.316599 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.316905 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.316885 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1daa908-9812-4ecf-923d-160fca8742c7-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vghr\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.317137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.317091 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.319351 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.319331 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr42f\" (UniqueName: \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-kube-api-access-mr42f\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-9vrgv\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.394935 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.394847 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.395093 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.395004 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.395093 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.395083 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbxmn\" (UniqueName: \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-kube-api-access-fbxmn\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.395275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.395138 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.395275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.395179 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.395275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.395230 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.395429 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.395390 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af543f1a-d698-4a5c-bf14-49e030f8b80e-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.395561 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.395467 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.396216 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.396188 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.412879 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.412846 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm"] Nov 01 22:34:44.418939 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.418903 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.432944 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.432911 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm"] Nov 01 22:34:44.471711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.471665 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:44.482079 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.482053 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:44.497164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497097 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxmn\" (UniqueName: \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-kube-api-access-fbxmn\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.497254 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497197 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.497254 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497244 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.497349 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497261 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlcf\" (UniqueName: \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-kube-api-access-cwlcf\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.497349 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497282 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.497516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497492 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.497560 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497527 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.497602 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497578 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.497684 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497653 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.497756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497702 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af543f1a-d698-4a5c-bf14-49e030f8b80e-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.497808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497761 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.497808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497788 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff42c4d5-6fe0-4a47-b371-238761f656dc-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.497917 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497824 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.497971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497936 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.498031 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.497986 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.498031 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.498019 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.498199 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.498176 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.498275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.498199 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.498344 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.498331 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.498412 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.498390 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.498468 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.498393 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.498550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.498530 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af543f1a-d698-4a5c-bf14-49e030f8b80e-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.498618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.498600 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.500538 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.500518 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.501048 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.501027 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.505163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.505078 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.505269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.505179 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbxmn\" (UniqueName: \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-kube-api-access-fbxmn\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.512308 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.511723 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk"] Nov 01 22:34:44.512308 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:34:44.511892 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e7ebcf_30e6_4c53_836e_07acb021ae73.slice/crio-980b521a91fc401dbb0940596a502ff514ccccccac4541e430b542a5c64b85cf WatchSource:0}: Error finding container 980b521a91fc401dbb0940596a502ff514ccccccac4541e430b542a5c64b85cf: Status 404 returned error can't find the container with id 980b521a91fc401dbb0940596a502ff514ccccccac4541e430b542a5c64b85cf Nov 01 22:34:44.514722 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.514688 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:44.514812 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.514759 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:44.514812 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.514786 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:44.601166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.599488 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.601166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.599585 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.601166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.599708 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.601166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.599740 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlcf\" (UniqueName: \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-kube-api-access-cwlcf\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.601166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.599778 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.601166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.599823 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.601166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.599924 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff42c4d5-6fe0-4a47-b371-238761f656dc-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.601166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.599991 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.601166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.600021 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.604282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.602510 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.604282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.604273 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff42c4d5-6fe0-4a47-b371-238761f656dc-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.605137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.604798 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.605137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.605049 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.605657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.605552 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.607379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.607329 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.609558 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.609418 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:44.610320 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.610297 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.611846 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.611802 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlcf\" (UniqueName: \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-kube-api-access-cwlcf\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.612670 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.612581 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.746374 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.746053 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:44.777709 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.777663 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr"] Nov 01 22:34:44.782800 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:34:44.782736 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1daa908_9812_4ecf_923d_160fca8742c7.slice/crio-c0fb4863a3ee5f0090a8bb186e437d540427ae7d76150145a421f4a465a411e2 WatchSource:0}: Error finding container c0fb4863a3ee5f0090a8bb186e437d540427ae7d76150145a421f4a465a411e2: Status 404 returned error can't find the container with id c0fb4863a3ee5f0090a8bb186e437d540427ae7d76150145a421f4a465a411e2 Nov 01 22:34:44.789643 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.789617 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv"] Nov 01 22:34:44.790844 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.790807 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:44.790965 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.790891 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:44.790965 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.790930 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:44.791223 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:34:44.791020 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131d0e2f_d0ba_4623_8bdb_e7b969c526db.slice/crio-36ac3f044b9909965d85b6ede61c50d85b00e036042a1b3874d073aab7464553 WatchSource:0}: Error finding container 36ac3f044b9909965d85b6ede61c50d85b00e036042a1b3874d073aab7464553: Status 404 returned error can't find the container with id 36ac3f044b9909965d85b6ede61c50d85b00e036042a1b3874d073aab7464553 Nov 01 22:34:44.793685 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.793655 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:44.793766 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.793730 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:44.793831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.793773 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:44.815098 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.815059 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k"] Nov 01 22:34:44.822839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.822818 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:44.841550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.841124 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-381659\"/\"default-dockercfg-x4b79\"" Nov 01 22:34:44.844853 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.844779 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-381659\"/\"kube-root-ca.crt\"" Nov 01 22:34:44.845471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.845198 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-381659\"/\"openshift-service-ca.crt\"" Nov 01 22:34:44.845471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.845308 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-381659\"/\"distinct-2-w1-381659-kserve-self-signed-certs\"" Nov 01 22:34:44.850180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.850096 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k"] Nov 01 22:34:44.908598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.908016 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tls-certs\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:44.908598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.908124 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-home\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:44.908598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.908194 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-dshm\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:44.908598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.908234 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tmp-dir\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:44.908598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.908261 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-model-cache\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:44.908598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.908407 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lhfs\" (UniqueName: \"kubernetes.io/projected/2bc77f6d-f23f-4191-bc0e-ba1abc195039-kube-api-access-8lhfs\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:44.933989 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.933047 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb"] Nov 01 22:34:44.936518 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:34:44.936494 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf543f1a_d698_4a5c_bf14_49e030f8b80e.slice/crio-0197542d9a7e8af418f4558d65d1fd5b8fbd790694b5b147e15d73a971811198 WatchSource:0}: Error finding container 0197542d9a7e8af418f4558d65d1fd5b8fbd790694b5b147e15d73a971811198: Status 404 returned error can't find the container with id 0197542d9a7e8af418f4558d65d1fd5b8fbd790694b5b147e15d73a971811198 Nov 01 22:34:44.939675 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.939014 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:44.939675 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.939090 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:44.939675 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:44.939173 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:45.029533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.017638 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-dshm\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.029533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.017701 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tmp-dir\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.029533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.017730 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-model-cache\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.029533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.017869 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lhfs\" (UniqueName: \"kubernetes.io/projected/2bc77f6d-f23f-4191-bc0e-ba1abc195039-kube-api-access-8lhfs\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.029533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.017991 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tls-certs\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.029533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.018075 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-home\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.029533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.018572 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-home\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.029533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.019195 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-model-cache\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.029533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.019436 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tmp-dir\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.029533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.022966 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tls-certs\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.029533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.024181 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-dshm\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.040329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.039852 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lhfs\" (UniqueName: \"kubernetes.io/projected/2bc77f6d-f23f-4191-bc0e-ba1abc195039-kube-api-access-8lhfs\") pod \"distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.145487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.117934 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm"] Nov 01 22:34:45.184597 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.184510 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:45.297085 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.290522 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:45.297085 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.290606 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:45.297085 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.290645 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:34:45.297085 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.293025 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" event={"ID":"ff42c4d5-6fe0-4a47-b371-238761f656dc","Type":"ContainerStarted","Data":"5fb002057170edac3a2e6ba0e2f488e083123289d28549a9b1261cf7c31febe0"} Nov 01 22:34:45.297085 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.295355 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" event={"ID":"af543f1a-d698-4a5c-bf14-49e030f8b80e","Type":"ContainerStarted","Data":"2f04610032a78ebc8a14ee695f03157874794ad482483e02c8e6e4477f5650c7"} Nov 01 22:34:45.297085 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.295410 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" event={"ID":"af543f1a-d698-4a5c-bf14-49e030f8b80e","Type":"ContainerStarted","Data":"0197542d9a7e8af418f4558d65d1fd5b8fbd790694b5b147e15d73a971811198"} Nov 01 22:34:45.335006 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.334945 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podStartSLOduration=1.334923295 podStartE2EDuration="1.334923295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:45.333737898 +0000 UTC m=+689.118176819" watchObservedRunningTime="2026-11-01 22:34:45.334923295 +0000 UTC m=+689.119362219" Nov 01 22:34:45.373159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.373062 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" event={"ID":"f1daa908-9812-4ecf-923d-160fca8742c7","Type":"ContainerStarted","Data":"151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263"} Nov 01 22:34:45.373591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.373480 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" event={"ID":"f1daa908-9812-4ecf-923d-160fca8742c7","Type":"ContainerStarted","Data":"c0fb4863a3ee5f0090a8bb186e437d540427ae7d76150145a421f4a465a411e2"} Nov 01 22:34:45.384944 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.376896 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" event={"ID":"131d0e2f-d0ba-4623-8bdb-e7b969c526db","Type":"ContainerStarted","Data":"035d24c71c565cb8d1404240ff5377a54b982c344fa13bb21d62df809ddcfcbd"} Nov 01 22:34:45.384944 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.376954 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" event={"ID":"131d0e2f-d0ba-4623-8bdb-e7b969c526db","Type":"ContainerStarted","Data":"36ac3f044b9909965d85b6ede61c50d85b00e036042a1b3874d073aab7464553"} Nov 01 22:34:45.394049 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.388852 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" event={"ID":"a7e7ebcf-30e6-4c53-836e-07acb021ae73","Type":"ContainerStarted","Data":"2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee"} Nov 01 22:34:45.394049 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.388920 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" event={"ID":"a7e7ebcf-30e6-4c53-836e-07acb021ae73","Type":"ContainerStarted","Data":"980b521a91fc401dbb0940596a502ff514ccccccac4541e430b542a5c64b85cf"} Nov 01 22:34:45.437922 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.437822 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podStartSLOduration=1.437803523 podStartE2EDuration="1.437803523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:45.413175128 +0000 UTC m=+689.197614050" watchObservedRunningTime="2026-11-01 22:34:45.437803523 +0000 UTC m=+689.222242445" Nov 01 22:34:45.476072 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.475398 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:45.493695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.491026 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:45.493993 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.492319 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:45.494230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.494185 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:45.498580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.498551 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:45.498774 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.498747 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:45.502673 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.502617 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podStartSLOduration=2.502596999 podStartE2EDuration="2.502596999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:45.439898503 +0000 UTC m=+689.224337425" watchObservedRunningTime="2026-11-01 22:34:45.502596999 +0000 UTC m=+689.287035921" Nov 01 22:34:45.512620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.510248 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podStartSLOduration=1.510229375 podStartE2EDuration="1.510229375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:45.487572514 +0000 UTC m=+689.272011448" watchObservedRunningTime="2026-11-01 22:34:45.510229375 +0000 UTC m=+689.294668297" Nov 01 22:34:45.610948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.610914 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:45.699601 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:45.699250 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k"] Nov 01 22:34:45.704278 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:34:45.704224 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bc77f6d_f23f_4191_bc0e_ba1abc195039.slice/crio-79d4b2504513ba778eed731a6b416c140ddb1ef012da7dd6734750831cfea163 WatchSource:0}: Error finding container 79d4b2504513ba778eed731a6b416c140ddb1ef012da7dd6734750831cfea163: Status 404 returned error can't find the container with id 79d4b2504513ba778eed731a6b416c140ddb1ef012da7dd6734750831cfea163 Nov 01 22:34:46.214298 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.214258 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:46.218590 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.218557 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:34:46.219132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.218935 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 22:34:46.522063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.519907 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:46.522063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.519970 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:46.522063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.520458 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:46.522063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.520495 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:46.574136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.552855 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" event={"ID":"ff42c4d5-6fe0-4a47-b371-238761f656dc","Type":"ContainerStarted","Data":"20e87c629602f35e99ec417a568d9676750596ab9dbac61019bd4b23d0ea005f"} Nov 01 22:34:46.596166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.578563 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podStartSLOduration=2.5785460909999998 podStartE2EDuration="2.578546091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:46.578255314 +0000 UTC m=+690.362694235" watchObservedRunningTime="2026-11-01 22:34:46.578546091 +0000 UTC m=+690.362985014" Nov 01 22:34:46.615964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.615765 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:46.615964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.615836 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:46.717132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.715941 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" event={"ID":"2bc77f6d-f23f-4191-bc0e-ba1abc195039","Type":"ContainerStarted","Data":"5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c"} Nov 01 22:34:46.717132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.715991 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" event={"ID":"2bc77f6d-f23f-4191-bc0e-ba1abc195039","Type":"ContainerStarted","Data":"79d4b2504513ba778eed731a6b416c140ddb1ef012da7dd6734750831cfea163"} Nov 01 22:34:46.752122 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.750424 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:46.773268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.772848 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:46.773268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:46.772905 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:47.189517 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:47.189482 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:47.224213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:47.224163 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:34:47.224869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:47.224806 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 22:34:47.478702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:47.478596 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:47.478970 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:47.478937 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:47.482494 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:47.482467 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:47.482679 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:47.482656 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:47.613140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:47.611241 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:47.613140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:47.611301 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:47.836193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:47.804633 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:47.836193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:47.804741 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:48.221131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:48.212879 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:34:48.221131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:48.212949 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 22:34:48.473698 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:48.473596 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:48.474399 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:48.474365 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:48.484165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:48.484101 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:48.484428 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:48.484401 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:48.792193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:48.788360 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:48.792193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:48.788432 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:48.792193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:48.788862 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:48.792193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:48.788901 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:49.310631 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:49.310073 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.310631 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:49.310168 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 22:34:49.645142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:49.644250 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.645142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:49.644326 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:49.645142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:49.644799 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.645142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:49.644840 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:49.747135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:49.745645 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.747135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:49.745703 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:49.768463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:49.761991 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.768463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:49.762045 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:49.840140 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:34:49.839308 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:34:50.220591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:50.216868 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:34:50.224126 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:50.220883 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 22:34:50.477190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:50.474651 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:50.477190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:50.474729 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:50.490668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:50.490632 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:50.490929 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:50.490905 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:50.733223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:50.707171 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:50.742138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:50.733778 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:50.855133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:50.824825 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:50.855133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:50.824889 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:51.253132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:51.252957 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:34:51.253352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:51.253204 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 22:34:51.473597 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:51.473545 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:51.473898 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:51.473852 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:51.491152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:51.485766 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:51.491152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:51.485824 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:51.698135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:51.697381 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:51.698135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:51.697441 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:51.763415 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:51.763380 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:51.763996 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:51.763961 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:52.259333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:52.259297 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:34:52.259613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:52.259586 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 22:34:52.481133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:52.479557 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:52.481133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:52.479621 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:52.500300 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:52.493495 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:52.500300 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:52.493553 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:52.610608 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:52.610513 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:52.610838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:52.610811 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:52.762135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:52.747599 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:52.773133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:52.766153 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:53.261308 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.261260 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:34:53.266136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.266071 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 22:34:53.664711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.556761 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:53.664711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.556837 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:53.664711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.557864 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:53.664711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.557908 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:53.743158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.716359 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:53.743158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.716424 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:53.792390 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.792333 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:53.792985 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.792949 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:53.854162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.853087 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podStartSLOduration=9.853063361 podStartE2EDuration="9.853063361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:46.767968996 +0000 UTC m=+690.552407917" watchObservedRunningTime="2026-11-01 22:34:53.853063361 +0000 UTC m=+697.637502283" Nov 01 22:34:53.924236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:53.900191 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq"] Nov 01 22:34:54.206185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.206088 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.217129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.213585 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:34:54.217393 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.217362 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 22:34:54.229360 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.229326 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b28a40\"/\"kube-root-ca.crt\"" Nov 01 22:34:54.233646 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.233619 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b28a40\"/\"openshift-service-ca.crt\"" Nov 01 22:34:54.243956 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.243729 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq"] Nov 01 22:34:54.258725 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.250234 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b28a40\"/\"distinct-2-w6-b28a40-kserve-self-signed-certs\"" Nov 01 22:34:54.274357 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.264817 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b28a40\"/\"default-dockercfg-6lslv\"" Nov 01 22:34:54.279230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.279181 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8txbl\" (UniqueName: \"kubernetes.io/projected/024bf9b0-d2dd-4499-a45b-043af8b73da2-kube-api-access-8txbl\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.300100 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.299558 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-tmp-dir\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.300411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.300376 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-dshm\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.300642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.300615 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-home\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.302364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.302333 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024bf9b0-d2dd-4499-a45b-043af8b73da2-tls-certs\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.302628 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.302600 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-model-cache\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.405152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.404000 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8txbl\" (UniqueName: \"kubernetes.io/projected/024bf9b0-d2dd-4499-a45b-043af8b73da2-kube-api-access-8txbl\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.405152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.404079 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-tmp-dir\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.405152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.404123 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-dshm\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.405152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.404182 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-home\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.405152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.404305 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024bf9b0-d2dd-4499-a45b-043af8b73da2-tls-certs\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.405152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.404372 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-model-cache\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.405152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.404878 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-model-cache\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.408009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.407605 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-home\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.421354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.421070 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-tmp-dir\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.561187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.482282 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:54.561187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.482353 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:54.561187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.483002 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:54.561187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.483045 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:54.561187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.533806 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024bf9b0-d2dd-4499-a45b-043af8b73da2-tls-certs\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.628273 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.616215 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:54.628273 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.616288 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:54.629410 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.629338 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8txbl\" (UniqueName: \"kubernetes.io/projected/024bf9b0-d2dd-4499-a45b-043af8b73da2-kube-api-access-8txbl\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.696838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.696756 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-dshm\") pod \"distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:54.765203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.757374 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:54.765203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.757432 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:54.954178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:54.948437 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:55.189165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.188423 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:55.304220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.263323 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:34:55.304220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.263388 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 22:34:55.530888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.506400 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:55.530888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.506484 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:55.531499 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.531373 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:55.531499 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.531438 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:55.688317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.669218 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:55.688317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.669281 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:55.783165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.772036 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:55.783165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.772411 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:55.783165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:55.772454 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:56.276055 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:56.276018 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:34:56.570978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:56.570882 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 22:34:56.590233 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:56.516928 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:56.590530 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:56.590498 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:56.590676 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:56.516977 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:56.590786 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:56.590768 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:56.610747 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:56.610708 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:56.611017 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:56.610978 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:56.758423 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:56.758389 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:56.758712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:56.758675 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:56.927130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:56.924757 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:34:57.559152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:57.538586 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:34:57.559152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:57.538656 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 01 22:34:57.559152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:57.538770 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:57.559152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:57.538796 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:57.667655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:57.642663 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:57.667655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:57.642730 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:57.758954 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:57.753872 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:34:57.758954 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:57.753937 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 01 22:34:57.844134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:57.841013 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:58.008192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:57.999044 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq"] Nov 01 22:34:58.217132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:58.199213 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" event={"ID":"024bf9b0-d2dd-4499-a45b-043af8b73da2","Type":"ContainerStarted","Data":"80f4ab4a4bd6e3124e249e245514067de50c6c03205a681f1fb473bd4a9c7265"} Nov 01 22:34:58.219390 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:58.219225 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:58.233191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:58.221120 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:58.233191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:58.223975 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:34:58.499132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:58.498851 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:34:58.499132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:58.498912 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 22:34:58.613132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:58.612556 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 22:34:58.613132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:58.612615 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 22:34:59.135137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.133001 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:59.135137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.133670 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:59.140129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.137326 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:34:59.185825 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.172974 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt"] Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.212457 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt"] Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.212599 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.231401 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.231484 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.231523 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddt6\" (UniqueName: \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-kube-api-access-fddt6\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.231555 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.231643 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.231736 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.231786 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.231826 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.231897 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.241128 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:59.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.243010 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:59.274031 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.269027 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" event={"ID":"024bf9b0-d2dd-4499-a45b-043af8b73da2","Type":"ContainerStarted","Data":"a0a0d219e3186e6aec22586d96212ab6d8d7f602330b3dbfdda1ef104f57f401"} Nov 01 22:34:59.274031 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.272595 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.379329 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.379459 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.379557 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.379600 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddt6\" (UniqueName: \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-kube-api-access-fddt6\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.379634 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.379786 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.379957 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.380063 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.380131 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.383028 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.386455 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.386955 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.387638 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.387900 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.398226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.390156 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.419973 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.416333 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.451094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.450843 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.452197 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.452028 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddt6\" (UniqueName: \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-kube-api-access-fddt6\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.481726 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.477683 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podStartSLOduration=6.477660188 podStartE2EDuration="6.477660188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:59.383534065 +0000 UTC m=+703.167972989" watchObservedRunningTime="2026-11-01 22:34:59.477660188 +0000 UTC m=+703.262099110" Nov 01 22:34:59.482602 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.482144 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:59.561958 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.561594 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:34:59.628747 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.626215 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:34:59.628747 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.626876 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:34:59.950270 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.950217 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:34:59.954148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:34:59.954080 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:35:00.160324 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:00.156691 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt"] Nov 01 22:35:00.180770 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:35:00.180733 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3f43b1_c00b_4e77_87cd_cba74f4ba1db.slice/crio-fc3964d23b6ac0328c9241a24d772e17962ef4ca7e76d584616ee081b9a55a85 WatchSource:0}: Error finding container fc3964d23b6ac0328c9241a24d772e17962ef4ca7e76d584616ee081b9a55a85: Status 404 returned error can't find the container with id fc3964d23b6ac0328c9241a24d772e17962ef4ca7e76d584616ee081b9a55a85 Nov 01 22:35:00.235740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:00.229270 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:00.235740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:00.229397 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:00.235740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:00.229452 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:00.292297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:00.292163 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" event={"ID":"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db","Type":"ContainerStarted","Data":"fc3964d23b6ac0328c9241a24d772e17962ef4ca7e76d584616ee081b9a55a85"} Nov 01 22:35:00.295744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:00.295232 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:35:00.295744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:00.295474 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:35:01.308637 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:01.308085 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" event={"ID":"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db","Type":"ContainerStarted","Data":"8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43"} Nov 01 22:35:01.627272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:01.627196 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:35:01.628931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:01.628906 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:35:01.629048 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:01.628962 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 22:35:02.627495 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:02.627457 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:35:02.627906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:02.627534 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 22:35:03.627333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:03.627296 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:35:03.627524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:03.627353 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 22:35:04.627666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:04.627631 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:35:04.628085 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:04.627686 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 22:35:04.960255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:04.960196 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:35:04.975234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:04.975208 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:35:04.995302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:04.995242 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" podStartSLOduration=5.9952210059999995 podStartE2EDuration="5.995221006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:01.361452781 +0000 UTC m=+705.145891704" watchObservedRunningTime="2026-11-01 22:35:04.995221006 +0000 UTC m=+708.779659928" Nov 01 22:35:05.474965 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:05.474929 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:35:05.627871 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:05.627831 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:35:05.628273 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:05.627905 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 22:35:06.627629 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:06.627591 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:35:06.627812 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:06.627658 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 22:35:07.644408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.643407 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:35:07.645243 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.645218 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:35:07.646403 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.646381 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:35:07.785867 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.785825 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx"] Nov 01 22:35:07.792386 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.792360 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:07.798329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.798297 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-00e49d\"/\"distinct-2-w3-00e49d-kserve-self-signed-certs\"" Nov 01 22:35:07.801978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.801927 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx"] Nov 01 22:35:07.918666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.918576 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c8j4\" (UniqueName: \"kubernetes.io/projected/9ed3436f-6092-45a6-b2df-5d59b28d43bf-kube-api-access-5c8j4\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:07.918666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.918644 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tmp-dir\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:07.918864 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.918821 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-model-cache\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:07.918922 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.918886 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-home\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:07.919008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.918989 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tls-certs\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:07.919073 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:07.919045 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-dshm\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.023128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.021184 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8j4\" (UniqueName: \"kubernetes.io/projected/9ed3436f-6092-45a6-b2df-5d59b28d43bf-kube-api-access-5c8j4\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.023128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.021280 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tmp-dir\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.023128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.021428 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-model-cache\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.023128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.021493 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-home\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.023128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.021564 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tls-certs\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.023128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.021610 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-dshm\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.023128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.022517 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-model-cache\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.023635 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.023194 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tmp-dir\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.023635 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.023433 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-home\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.031130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.028332 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tls-certs\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.046147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.036063 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c8j4\" (UniqueName: \"kubernetes.io/projected/9ed3436f-6092-45a6-b2df-5d59b28d43bf-kube-api-access-5c8j4\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.046147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.039042 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-dshm\") pod \"distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.112131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.111687 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:08.429786 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.429748 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx"] Nov 01 22:35:08.431338 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:35:08.431312 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ed3436f_6092_45a6_b2df_5d59b28d43bf.slice/crio-2afb237680ca83922cdf992638a0623ac6e7a224df3240340ffe5eef54a9b48a WatchSource:0}: Error finding container 2afb237680ca83922cdf992638a0623ac6e7a224df3240340ffe5eef54a9b48a: Status 404 returned error can't find the container with id 2afb237680ca83922cdf992638a0623ac6e7a224df3240340ffe5eef54a9b48a Nov 01 22:35:08.482968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:08.482939 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" event={"ID":"9ed3436f-6092-45a6-b2df-5d59b28d43bf","Type":"ContainerStarted","Data":"2afb237680ca83922cdf992638a0623ac6e7a224df3240340ffe5eef54a9b48a"} Nov 01 22:35:09.491089 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:09.491051 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" event={"ID":"9ed3436f-6092-45a6-b2df-5d59b28d43bf","Type":"ContainerStarted","Data":"0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9"} Nov 01 22:35:09.509512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:09.509465 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podStartSLOduration=2.509450715 podStartE2EDuration="2.509450715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:09.507533217 +0000 UTC m=+713.291972141" watchObservedRunningTime="2026-11-01 22:35:09.509450715 +0000 UTC m=+713.293889655" Nov 01 22:35:10.112841 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:10.112804 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:14.365147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.354015 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6"] Nov 01 22:35:14.472948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.472896 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.565133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.560632 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.565133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.560750 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.565133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.560786 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.565133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.560866 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.565133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.561019 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.565133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.561063 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ltn9\" (UniqueName: \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-kube-api-access-5ltn9\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.565133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.561140 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fade718-419b-4d7a-9dd9-417cb6ec2621-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.565133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.561196 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.565133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.561239 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.689255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.689089 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.689582 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.689554 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.689739 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.689709 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.689859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.689841 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.690052 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.690019 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.690351 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.690332 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.690519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.690501 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ltn9\" (UniqueName: \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-kube-api-access-5ltn9\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.690677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.690648 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fade718-419b-4d7a-9dd9-417cb6ec2621-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.690854 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.690822 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.691025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.691000 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.693665 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.693640 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.722198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.711457 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.724075 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.724035 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.734801 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.734767 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fade718-419b-4d7a-9dd9-417cb6ec2621-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.859131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.840024 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6"] Nov 01 22:35:14.969410 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.969325 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z"] Nov 01 22:35:14.989132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.959965 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:14.989132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:14.960495 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:15.024230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.011287 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z"] Nov 01 22:35:15.024230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.011358 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm"] Nov 01 22:35:15.024230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.017242 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.093141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.075196 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqlq\" (UniqueName: \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-kube-api-access-csqlq\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.093479 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.093448 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.093621 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.093604 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.093854 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.093835 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.102133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.094044 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.102493 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.102459 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.102652 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.102633 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.107143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.102875 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.107401 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.107373 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.116136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.078469 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:15.145585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.135260 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ltn9\" (UniqueName: \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-kube-api-access-5ltn9\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:15.145827 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.135527 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm"] Nov 01 22:35:15.146011 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.145992 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2"] Nov 01 22:35:15.182157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.135633 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.208768 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.208728 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.208913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.208787 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.208913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.208872 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.208913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.208904 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqlq\" (UniqueName: \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-kube-api-access-csqlq\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.209057 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.208944 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.209057 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.208968 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.209196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209064 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.209196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209096 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.209196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209148 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.209348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209215 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.209348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209248 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.209348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209280 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.209348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209341 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.209516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209381 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.209516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209410 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.209516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209471 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.209516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209507 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65bjm\" (UniqueName: \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-kube-api-access-65bjm\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.209697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.209571 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.218135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.210031 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.218135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.212865 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.218135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.213367 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.264194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.237840 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.264194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.238092 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.264194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.241361 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2"] Nov 01 22:35:15.264194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.241479 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.349131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.345432 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqlq\" (UniqueName: \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-kube-api-access-csqlq\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.349131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.347924 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.372130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.363368 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:15.393128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.391692 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.467600 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.440555 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.521580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.521474 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:35:15.522396 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.522361 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.525336 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.525310 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.597133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.573278 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.597133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.581266 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.597133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.527924 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:15.597133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.587406 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e25203e8-2d60-4abd-93ec-49a19a6000bc-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.597133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.592245 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.597133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.592311 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.614133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.599967 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.601711 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.625323 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.634223 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.634289 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.634336 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.634375 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.634419 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.634491 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.634595 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.634631 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.634680 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bjm\" (UniqueName: \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-kube-api-access-65bjm\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.634710 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfm5k\" (UniqueName: \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-kube-api-access-mfm5k\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.640132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.635341 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.658515 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.656864 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.689471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.689435 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.710601 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.697896 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65bjm\" (UniqueName: \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-kube-api-access-65bjm\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.710817 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.709721 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.737643 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.737600 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.737839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.737747 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.737839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.737799 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfm5k\" (UniqueName: \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-kube-api-access-mfm5k\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.737987 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.737966 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.738131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.738094 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.738192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.738175 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e25203e8-2d60-4abd-93ec-49a19a6000bc-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.738288 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.738271 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.738345 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.738328 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.738399 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.738372 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.738953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.738916 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.742136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.741882 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.743574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.743549 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.744405 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.744382 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e25203e8-2d60-4abd-93ec-49a19a6000bc-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.744811 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.744782 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.745681 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.745650 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.766344 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.766257 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.769452 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.769403 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfm5k\" (UniqueName: \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-kube-api-access-mfm5k\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.805576 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.803690 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.852683 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.849502 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.931585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.931543 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:15.957519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.956475 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:15.958320 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:15.957830 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:16.235438 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.235401 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6"] Nov 01 22:35:16.239057 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.239015 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.239201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.239136 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.239201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.239188 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.335345 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.335312 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z"] Nov 01 22:35:16.338550 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:35:16.338514 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdd3ac7_20a2_4844_a99f_6304dc4a5bb4.slice/crio-8fe312bb84a32e7a0aa42d3ef7c7920610cf08ce74f4e4af6769aa13e54ddd93 WatchSource:0}: Error finding container 8fe312bb84a32e7a0aa42d3ef7c7920610cf08ce74f4e4af6769aa13e54ddd93: Status 404 returned error can't find the container with id 8fe312bb84a32e7a0aa42d3ef7c7920610cf08ce74f4e4af6769aa13e54ddd93 Nov 01 22:35:16.341932 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.341899 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.342032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.341972 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.342032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.342003 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.374168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.374141 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2"] Nov 01 22:35:16.379283 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:35:16.379255 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25203e8_2d60_4abd_93ec_49a19a6000bc.slice/crio-a838d49d223f88f5d2db50c0532a1ddac9bacdb3e61a58193b0c06242daa4c22 WatchSource:0}: Error finding container a838d49d223f88f5d2db50c0532a1ddac9bacdb3e61a58193b0c06242daa4c22: Status 404 returned error can't find the container with id a838d49d223f88f5d2db50c0532a1ddac9bacdb3e61a58193b0c06242daa4c22 Nov 01 22:35:16.382430 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.382394 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.382566 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.382475 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.382566 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.382507 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.436776 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.436720 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm"] Nov 01 22:35:16.474310 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.474256 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.474605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.474580 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.474752 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.474732 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:16.766162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.766101 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" event={"ID":"5fade718-419b-4d7a-9dd9-417cb6ec2621","Type":"ContainerStarted","Data":"c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088"} Nov 01 22:35:16.766320 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.766194 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" event={"ID":"5fade718-419b-4d7a-9dd9-417cb6ec2621","Type":"ContainerStarted","Data":"c8ec0ce0ef57942ecb8c866a4605171b6ff0d485c406747e7a4cf248dd686d09"} Nov 01 22:35:16.770160 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.769512 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" event={"ID":"51a87e58-fa5f-4806-81cd-bed0b1e62ece","Type":"ContainerStarted","Data":"281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b"} Nov 01 22:35:16.770160 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.769571 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" event={"ID":"51a87e58-fa5f-4806-81cd-bed0b1e62ece","Type":"ContainerStarted","Data":"26465d2338bc2879481c0dcbfca11b060645b14e855279b1399f2089f0b54eee"} Nov 01 22:35:16.776038 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.775155 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" event={"ID":"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4","Type":"ContainerStarted","Data":"512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55"} Nov 01 22:35:16.776038 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.775214 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" event={"ID":"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4","Type":"ContainerStarted","Data":"8fe312bb84a32e7a0aa42d3ef7c7920610cf08ce74f4e4af6769aa13e54ddd93"} Nov 01 22:35:16.779571 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.778742 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" event={"ID":"e25203e8-2d60-4abd-93ec-49a19a6000bc","Type":"ContainerStarted","Data":"b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484"} Nov 01 22:35:16.779571 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.778792 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" event={"ID":"e25203e8-2d60-4abd-93ec-49a19a6000bc","Type":"ContainerStarted","Data":"a838d49d223f88f5d2db50c0532a1ddac9bacdb3e61a58193b0c06242daa4c22"} Nov 01 22:35:16.800498 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.800424 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podStartSLOduration=3.800401416 podStartE2EDuration="3.800401416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:16.792950747 +0000 UTC m=+720.577389669" watchObservedRunningTime="2026-11-01 22:35:16.800401416 +0000 UTC m=+720.584840339" Nov 01 22:35:16.855527 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.855419 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" podStartSLOduration=2.8554005399999998 podStartE2EDuration="2.85540054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:16.848863548 +0000 UTC m=+720.633302480" watchObservedRunningTime="2026-11-01 22:35:16.85540054 +0000 UTC m=+720.639839461" Nov 01 22:35:16.891250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.891197 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" podStartSLOduration=2.891181316 podStartE2EDuration="2.891181316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:16.888884043 +0000 UTC m=+720.673322975" watchObservedRunningTime="2026-11-01 22:35:16.891181316 +0000 UTC m=+720.675620237" Nov 01 22:35:16.963145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.957478 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:16.963145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:16.958323 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:17.413254 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:17.412464 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:17.422788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:17.422376 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:35:17.422788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:17.422463 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 22:35:17.542158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:17.538763 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:17.542158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:17.538933 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:35:17.542158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:17.538977 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 22:35:18.014256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.014216 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:35:18.020132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.014630 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 22:35:18.020132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.015000 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:35:18.020132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.015039 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 22:35:18.140285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.123675 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:18.550009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.432259 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:35:18.550009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.432356 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 22:35:18.550009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.533933 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:35:18.550009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.534007 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 22:35:18.616215 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.616179 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:18.719135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.706008 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podStartSLOduration=4.705985758 podStartE2EDuration="4.705985758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:16.928982928 +0000 UTC m=+720.713421850" watchObservedRunningTime="2026-11-01 22:35:18.705985758 +0000 UTC m=+722.490424681" Nov 01 22:35:18.769270 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.769211 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:18.985771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.985729 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:35:18.986040 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.986014 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 22:35:18.993132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.989545 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:35:18.993386 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:18.993357 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 22:35:19.428816 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:19.428771 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:35:19.429095 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:19.429069 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 22:35:19.504402 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:35:19.504331 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:35:19.532508 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:19.532474 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:35:19.532823 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:19.532774 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 22:35:19.939132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:19.935715 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:19.975174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:19.968431 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:35:19.975174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:19.968489 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 22:35:19.975174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:19.971518 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:35:19.975174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:19.971573 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 22:35:20.008854 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:20.008804 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:20.444131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:20.443767 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:35:20.444131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:20.443832 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 22:35:20.469270 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:20.469220 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:20.542146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:20.530687 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:35:20.542146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:20.530753 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 22:35:20.764517 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:20.764426 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:20.958130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:20.957891 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:35:20.958130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:20.957956 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 22:35:20.958712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:20.958620 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:35:20.958712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:20.958657 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 22:35:21.012138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:21.011202 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:21.012138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:21.011663 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:35:21.447138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:21.443271 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:35:21.447138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:21.443337 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 22:35:21.472166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:21.472101 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:21.492620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:21.492586 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:35:21.554134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:21.550475 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:35:21.554134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:21.550538 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 22:35:22.043181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.010678 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:22.050136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.048278 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.050136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.048333 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 22:35:22.050136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.048758 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:22.109133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.108268 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.109133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.108341 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 22:35:22.472302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.472259 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.472688 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.472656 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 22:35:22.560135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.534231 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.692171 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.562660 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:22.692171 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.624674 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 22:35:22.815414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.815312 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:22.953086 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.953049 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.953400 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.953370 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 22:35:22.962132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.960846 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.962132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:22.960924 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 22:35:23.039134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.038195 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:23.039134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.038945 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:23.051277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.051237 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:23.413892 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.413855 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:35:23.414247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.414196 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 22:35:23.551131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.542737 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:35:23.551131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.542800 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 22:35:23.628246 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.628198 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:35:23.960615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.960574 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:35:23.960906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.960874 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 22:35:23.961760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.961738 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:35:23.961952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:23.961930 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 22:35:24.045716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.045664 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:24.072245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.072194 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:35:24.439258 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.438741 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:35:24.439258 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.438802 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 22:35:24.599307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.547655 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:35:24.599307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.547712 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 22:35:24.647266 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.647220 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:24.733343 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.733256 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:24.733958 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.733476 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:35:24.733958 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.733512 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:35:24.989231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.967409 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:35:24.989231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.988296 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 22:35:24.989231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.989011 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:35:24.989231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:24.989056 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 22:35:25.410411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:25.410364 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:35:25.410939 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:25.410433 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 22:35:25.529523 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:25.529487 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:35:25.529745 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:25.529549 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 01 22:35:25.960617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:25.960581 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:35:25.960919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:25.960889 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 22:35:26.773235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.773198 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:26.774723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.774694 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:26.776772 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.776749 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:35:26.784478 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.784455 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:26.786287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.786266 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:26.786957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.786587 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:26.789205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.789191 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:26.790526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.790508 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:35:26.795210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.793481 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:35:26.985564 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.985531 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:26.986711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.986685 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:26.988414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:26.988393 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:35:35.678409 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:35.678326 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:35.777259 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:35.777212 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:35.954649 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:35.954221 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.052266 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:36.052215 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:35:36.070264 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:36.070217 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.213302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:36.213203 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.762577 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:36.761702 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:35:36.823760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:36.802945 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:35:36.823760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:36.812904 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.867748 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:36.865432 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:35:36.959257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:36.937884 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Nov 01 22:35:37.142200 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:37.120571 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:37.142200 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:37.126016 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:35:37.175911 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:37.175877 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:37.187019 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:37.186990 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:35:37.576185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:37.571318 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:35:37.576185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:37.571395 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" podUID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:37.917526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:37.917481 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.039216 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:38.010669 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:35:38.155136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:38.128270 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.180195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:38.155990 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.686244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:38.673601 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:35:38.686244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:38.673717 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.712135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:38.708322 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:35:38.712135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:38.708408 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.012200 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:39.011483 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.157274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:39.069397 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.157274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:39.102996 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:35:39.157274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:39.103045 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.157274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:39.124527 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.157274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:39.139159 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.184134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:39.183337 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.636134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:39.635231 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:35:39.636134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:39.635301 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:40.086634 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:40.086528 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:40.108897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:40.108345 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:40.180233 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:40.180187 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:35:40.226132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:40.199248 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:35:40.885718 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:40.883275 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:35:40.885718 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:40.883358 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:41.131137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:41.127901 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:41.148221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:41.131710 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:35:41.193146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:41.191710 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:41.244235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:41.244183 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:41.928994 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:41.928966 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:35:42.196457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:42.196364 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:35:42.200309 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:42.200283 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:35:42.272849 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:42.272803 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:35:49.031944 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:49.031909 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-6304e6"] Nov 01 22:35:49.037465 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:49.037446 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-6304e6" Nov 01 22:35:49.041623 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:49.041596 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vrfk9\"" Nov 01 22:35:49.043861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:49.043792 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-6304e6"] Nov 01 22:35:49.156659 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:49.156620 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm48f\" (UniqueName: \"kubernetes.io/projected/a1564123-1c54-4661-b3b8-182d12fa9e0a-kube-api-access-gm48f\") pod \"test-curl-7024-6304e6\" (UID: \"a1564123-1c54-4661-b3b8-182d12fa9e0a\") " pod="opendatahub/test-curl-7024-6304e6" Nov 01 22:35:49.258158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:49.258120 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm48f\" (UniqueName: \"kubernetes.io/projected/a1564123-1c54-4661-b3b8-182d12fa9e0a-kube-api-access-gm48f\") pod \"test-curl-7024-6304e6\" (UID: \"a1564123-1c54-4661-b3b8-182d12fa9e0a\") " pod="opendatahub/test-curl-7024-6304e6" Nov 01 22:35:49.266426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:49.266394 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm48f\" (UniqueName: \"kubernetes.io/projected/a1564123-1c54-4661-b3b8-182d12fa9e0a-kube-api-access-gm48f\") pod \"test-curl-7024-6304e6\" (UID: \"a1564123-1c54-4661-b3b8-182d12fa9e0a\") " pod="opendatahub/test-curl-7024-6304e6" Nov 01 22:35:49.282470 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:35:49.282431 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:35:49.365785 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:49.365691 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-6304e6" Nov 01 22:35:49.710039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:49.710010 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-6304e6"] Nov 01 22:35:49.711332 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:35:49.711306 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1564123_1c54_4661_b3b8_182d12fa9e0a.slice/crio-8c8c9102463471acc8197990590b8d8aed3dc4b5143af7e23775b8433a695ada WatchSource:0}: Error finding container 8c8c9102463471acc8197990590b8d8aed3dc4b5143af7e23775b8433a695ada: Status 404 returned error can't find the container with id 8c8c9102463471acc8197990590b8d8aed3dc4b5143af7e23775b8433a695ada Nov 01 22:35:50.528437 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:50.528393 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-6304e6" event={"ID":"a1564123-1c54-4661-b3b8-182d12fa9e0a","Type":"ContainerStarted","Data":"8c8c9102463471acc8197990590b8d8aed3dc4b5143af7e23775b8433a695ada"} Nov 01 22:35:51.536487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:51.536456 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-6304e6" event={"ID":"a1564123-1c54-4661-b3b8-182d12fa9e0a","Type":"ContainerStarted","Data":"9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d"} Nov 01 22:35:51.558595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:51.558542 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-6304e6" podStartSLOduration=1.59611136 podStartE2EDuration="2.558527933s" totalImagesPullingTime="962.416573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:51.554326287 +0000 UTC m=+755.338765208" watchObservedRunningTime="2026-11-01 22:35:51.558527933 +0000 UTC m=+755.342966853" Nov 01 22:35:52.228418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.228389 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-6304e6"] Nov 01 22:35:52.251824 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.251783 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-6304e6"] Nov 01 22:35:52.376275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.376242 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-1646ac"] Nov 01 22:35:52.376919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.376904 6466 cpu_manager.go:414] "containerMap: removing container" podUID="a1564123-1c54-4661-b3b8-182d12fa9e0a" containerName="test-curl-7024-6304e6" Nov 01 22:35:52.376971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.376920 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1564123-1c54-4661-b3b8-182d12fa9e0a" containerName="test-curl-7024-6304e6" Nov 01 22:35:52.377043 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.377034 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1564123-1c54-4661-b3b8-182d12fa9e0a" containerName="test-curl-7024-6304e6" Nov 01 22:35:52.380509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.380493 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-1646ac" Nov 01 22:35:52.387939 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.387912 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-1646ac"] Nov 01 22:35:52.470792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.470760 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2qzx\" (UniqueName: \"kubernetes.io/projected/e919d088-f9d2-4753-b49b-4c721ca8255b-kube-api-access-p2qzx\") pod \"test-curl-7024-1646ac\" (UID: \"e919d088-f9d2-4753-b49b-4c721ca8255b\") " pod="opendatahub/test-curl-7024-1646ac" Nov 01 22:35:52.572232 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.572155 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2qzx\" (UniqueName: \"kubernetes.io/projected/e919d088-f9d2-4753-b49b-4c721ca8255b-kube-api-access-p2qzx\") pod \"test-curl-7024-1646ac\" (UID: \"e919d088-f9d2-4753-b49b-4c721ca8255b\") " pod="opendatahub/test-curl-7024-1646ac" Nov 01 22:35:52.581468 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.581445 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2qzx\" (UniqueName: \"kubernetes.io/projected/e919d088-f9d2-4753-b49b-4c721ca8255b-kube-api-access-p2qzx\") pod \"test-curl-7024-1646ac\" (UID: \"e919d088-f9d2-4753-b49b-4c721ca8255b\") " pod="opendatahub/test-curl-7024-1646ac" Nov 01 22:35:52.713173 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:52.712463 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-1646ac" Nov 01 22:35:53.077394 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:53.077365 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-1646ac"] Nov 01 22:35:53.077784 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:35:53.077755 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode919d088_f9d2_4753_b49b_4c721ca8255b.slice/crio-bb6e43cdba5aaedc7892f5568c8b769dc248f48085d34ab3b2c9f02558001f2d WatchSource:0}: Error finding container bb6e43cdba5aaedc7892f5568c8b769dc248f48085d34ab3b2c9f02558001f2d: Status 404 returned error can't find the container with id bb6e43cdba5aaedc7892f5568c8b769dc248f48085d34ab3b2c9f02558001f2d Nov 01 22:35:53.552329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:53.552295 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-1646ac" event={"ID":"e919d088-f9d2-4753-b49b-4c721ca8255b","Type":"ContainerStarted","Data":"cd58530aa6e0cbe4359c18b55905343a96e6fedf6f5810dffa92e64f3050509b"} Nov 01 22:35:53.552329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:53.552338 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-1646ac" event={"ID":"e919d088-f9d2-4753-b49b-4c721ca8255b","Type":"ContainerStarted","Data":"bb6e43cdba5aaedc7892f5568c8b769dc248f48085d34ab3b2c9f02558001f2d"} Nov 01 22:35:53.552594 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:53.552556 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-6304e6" podUID="a1564123-1c54-4661-b3b8-182d12fa9e0a" containerName="test-curl-7024-6304e6" containerID="cri-o://9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d" gracePeriod=2 Nov 01 22:35:53.571035 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:53.570987 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-1646ac" podStartSLOduration=1.570973617 podStartE2EDuration="1.570973617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:53.567360644 +0000 UTC m=+757.351799562" watchObservedRunningTime="2026-11-01 22:35:53.570973617 +0000 UTC m=+757.355412537" Nov 01 22:35:54.206604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:54.206570 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-1646ac"] Nov 01 22:35:54.215532 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:54.215501 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-1646ac"] Nov 01 22:35:55.566324 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.566264 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-1646ac" podUID="e919d088-f9d2-4753-b49b-4c721ca8255b" containerName="test-curl-7024-1646ac" containerID="cri-o://cd58530aa6e0cbe4359c18b55905343a96e6fedf6f5810dffa92e64f3050509b" gracePeriod=2 Nov 01 22:35:55.698942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.698906 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8"] Nov 01 22:35:55.699887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.699865 6466 cpu_manager.go:414] "containerMap: removing container" podUID="e919d088-f9d2-4753-b49b-4c721ca8255b" containerName="test-curl-7024-1646ac" Nov 01 22:35:55.699887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.699887 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e919d088-f9d2-4753-b49b-4c721ca8255b" containerName="test-curl-7024-1646ac" Nov 01 22:35:55.700165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.700005 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e919d088-f9d2-4753-b49b-4c721ca8255b" containerName="test-curl-7024-1646ac" Nov 01 22:35:55.704074 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.704047 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.714504 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.714479 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0500c01f-openshift-default-dockercfg-hw9kd\"" Nov 01 22:35:55.732405 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.732366 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-data\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.732645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.732427 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-socket\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.732645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.732524 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-credential-socket\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.732645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.732611 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-token\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.732968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.732648 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28632ab8-53e2-4244-82e1-e0daa463be0d-istiod-ca-cert\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.732968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.732745 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-certs\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.732968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.732818 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-envoy\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.732968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.732908 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-podinfo\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.733302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.733003 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lkqb\" (UniqueName: \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-kube-api-access-4lkqb\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.734301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.734274 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8"] Nov 01 22:35:55.860288 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.860207 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-6304e6" Nov 01 22:35:55.869326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.865563 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-credential-socket\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.876312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.876279 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-token\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.907146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.894320 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28632ab8-53e2-4244-82e1-e0daa463be0d-istiod-ca-cert\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.907146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.894492 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-certs\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.907146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.894624 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-envoy\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.907146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.894770 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-podinfo\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.907146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.894919 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lkqb\" (UniqueName: \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-kube-api-access-4lkqb\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.907146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.894950 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-data\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.907146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.895012 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-socket\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.907146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.865967 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-credential-socket\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.931160 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.925781 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-podinfo\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.931160 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.926082 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-certs\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.944490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.944458 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-envoy\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.971132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.958420 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-data\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.971132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.958502 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-socket\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.971132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.958970 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28632ab8-53e2-4244-82e1-e0daa463be0d-istiod-ca-cert\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.993997 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.993948 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-token\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:55.998236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:55.998202 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1564123-1c54-4661-b3b8-182d12fa9e0a-kube-api-access-gm48f\" (UniqueName: \"kubernetes.io/projected/a1564123-1c54-4661-b3b8-182d12fa9e0a-kube-api-access-gm48f\") pod \"a1564123-1c54-4661-b3b8-182d12fa9e0a\" (UID: \"a1564123-1c54-4661-b3b8-182d12fa9e0a\") " Nov 01 22:35:56.028169 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.026509 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1564123-1c54-4661-b3b8-182d12fa9e0a-kube-api-access-gm48f" pod "a1564123-1c54-4661-b3b8-182d12fa9e0a" (UID: "a1564123-1c54-4661-b3b8-182d12fa9e0a"). InnerVolumeSpecName "kube-api-access-gm48f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:56.041133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.039821 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lkqb\" (UniqueName: \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-kube-api-access-4lkqb\") pod \"e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:56.102390 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.101206 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm48f\" (UniqueName: \"kubernetes.io/projected/a1564123-1c54-4661-b3b8-182d12fa9e0a-kube-api-access-gm48f\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:56.323727 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.323688 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:56.344443 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.344414 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1564123-1c54-4661-b3b8-182d12fa9e0a" path="/var/lib/kubelet/pods/a1564123-1c54-4661-b3b8-182d12fa9e0a/volumes" Nov 01 22:35:56.640705 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.640661 6466 generic.go:356] "Generic (PLEG): container finished" podID="a1564123-1c54-4661-b3b8-182d12fa9e0a" containerID="9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d" exitCode=137 Nov 01 22:35:56.640705 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.640709 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-6304e6" Nov 01 22:35:56.641221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.640751 6466 scope.go:122] "RemoveContainer" containerID="9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d" Nov 01 22:35:56.653505 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.653484 6466 scope.go:122] "RemoveContainer" containerID="9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d" Nov 01 22:35:56.653861 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:35:56.653842 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d\": container with ID starting with 9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d not found: ID does not exist" containerID="9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d" Nov 01 22:35:56.653941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.653875 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d"} err="failed to get container status \"9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d\": rpc error: code = NotFound desc = could not find container \"9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d\": container with ID starting with 9918a1d0c012c09a94589708d30b955d901293b427e88aea08b35bcd37c8bb4d not found: ID does not exist" Nov 01 22:35:56.654954 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.654935 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8"] Nov 01 22:35:56.655267 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:35:56.655247 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28632ab8_53e2_4244_82e1_e0daa463be0d.slice/crio-64ead3e24798fbe641ee1de186292ae7ed03065cf6ae05982b68d82d7fde5cf2 WatchSource:0}: Error finding container 64ead3e24798fbe641ee1de186292ae7ed03065cf6ae05982b68d82d7fde5cf2: Status 404 returned error can't find the container with id 64ead3e24798fbe641ee1de186292ae7ed03065cf6ae05982b68d82d7fde5cf2 Nov 01 22:35:56.658095 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.658063 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:56.658260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.658154 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:56.658260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:56.658198 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:57.651298 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:57.651267 6466 generic.go:356] "Generic (PLEG): container finished" podID="e919d088-f9d2-4753-b49b-4c721ca8255b" containerID="cd58530aa6e0cbe4359c18b55905343a96e6fedf6f5810dffa92e64f3050509b" exitCode=137 Nov 01 22:35:57.653162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:57.653092 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" event={"ID":"28632ab8-53e2-4244-82e1-e0daa463be0d","Type":"ContainerStarted","Data":"70112b71f1a9c8a747cbfd6ffbe4e2ede4c9cb3fa726b16693178841760fea6e"} Nov 01 22:35:57.653294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:57.653191 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" event={"ID":"28632ab8-53e2-4244-82e1-e0daa463be0d","Type":"ContainerStarted","Data":"64ead3e24798fbe641ee1de186292ae7ed03065cf6ae05982b68d82d7fde5cf2"} Nov 01 22:35:57.679844 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:57.679756 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" podStartSLOduration=2.679743019 podStartE2EDuration="2.679743019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:57.675215939 +0000 UTC m=+761.459654862" watchObservedRunningTime="2026-11-01 22:35:57.679743019 +0000 UTC m=+761.464181941" Nov 01 22:35:57.760220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:57.760193 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-1646ac" Nov 01 22:35:57.949296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:57.948680 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e919d088-f9d2-4753-b49b-4c721ca8255b-kube-api-access-p2qzx\" (UniqueName: \"kubernetes.io/projected/e919d088-f9d2-4753-b49b-4c721ca8255b-kube-api-access-p2qzx\") pod \"e919d088-f9d2-4753-b49b-4c721ca8255b\" (UID: \"e919d088-f9d2-4753-b49b-4c721ca8255b\") " Nov 01 22:35:57.955338 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:57.955310 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e919d088-f9d2-4753-b49b-4c721ca8255b-kube-api-access-p2qzx" pod "e919d088-f9d2-4753-b49b-4c721ca8255b" (UID: "e919d088-f9d2-4753-b49b-4c721ca8255b"). InnerVolumeSpecName "kube-api-access-p2qzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:58.050982 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:58.050949 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2qzx\" (UniqueName: \"kubernetes.io/projected/e919d088-f9d2-4753-b49b-4c721ca8255b-kube-api-access-p2qzx\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:58.324425 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:58.324344 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:58.330103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:58.330077 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:58.342953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:58.342924 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e919d088-f9d2-4753-b49b-4c721ca8255b" path="/var/lib/kubelet/pods/e919d088-f9d2-4753-b49b-4c721ca8255b/volumes" Nov 01 22:35:58.663121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:58.663084 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-1646ac" Nov 01 22:35:58.663585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:58.663170 6466 scope.go:122] "RemoveContainer" containerID="cd58530aa6e0cbe4359c18b55905343a96e6fedf6f5810dffa92e64f3050509b" Nov 01 22:35:58.663585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:58.663412 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:58.664534 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:58.664514 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:35:59.237206 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.237176 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2"] Nov 01 22:35:59.243467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.243436 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.261641 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.261603 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2"] Nov 01 22:35:59.270447 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.270416 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.270602 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.270556 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.270671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.270651 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdgpr\" (UniqueName: \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-kube-api-access-fdgpr\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.270728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.270708 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.270801 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.270787 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.270919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.270894 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.271013 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.270987 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.271177 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.271067 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.271177 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.271156 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373047 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373172 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373230 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdgpr\" (UniqueName: \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-kube-api-access-fdgpr\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373280 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373300 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373442 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373358 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373442 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373409 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373537 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373456 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373625 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373598 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373681 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373651 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373681 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373671 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373797 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.373969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.373942 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.374294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.374273 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.375613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.375593 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.375613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.375604 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.386909 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.386879 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.387019 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.386947 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdgpr\" (UniqueName: \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-kube-api-access-fdgpr\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.578190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.574413 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:35:59.743723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.743682 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck"] Nov 01 22:35:59.748867 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.748845 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.769815 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.769761 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck"] Nov 01 22:35:59.784285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.784214 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.784475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.784336 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.784475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.784378 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.784594 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.784509 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.784594 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.784559 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.784724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.784685 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.784838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.784819 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.784931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.784912 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8rx\" (UniqueName: \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-kube-api-access-vp8rx\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.785007 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.784991 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.804175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.804137 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt"] Nov 01 22:35:59.810180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.809858 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.842195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.840079 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt"] Nov 01 22:35:59.846912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.846171 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt"] Nov 01 22:35:59.846912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.846294 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.854705 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.854208 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt"] Nov 01 22:35:59.887008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.886978 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.887247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887035 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.887247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887214 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.887335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887260 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.887335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887293 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.887411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887353 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.887485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887463 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.887562 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887538 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.887654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887638 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.887693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887672 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xtjl\" (UniqueName: \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-kube-api-access-6xtjl\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.887730 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887700 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.887730 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887721 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.887826 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887810 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.887862 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887838 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.887915 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887904 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8rx\" (UniqueName: \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-kube-api-access-vp8rx\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.887971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.887943 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.888082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888067 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.888182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888103 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.888246 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888193 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.888318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888260 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.888318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888302 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.888414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888367 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.888414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888402 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.888485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888448 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29n8t\" (UniqueName: \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-kube-api-access-29n8t\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.888525 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888511 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.888568 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888530 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.888612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888584 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.888612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888605 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8dc27f4-6212-4492-a1d5-dcced032e3af-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.888784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888762 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.888847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.888755 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.889184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.889098 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.889361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.889342 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.890378 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.890359 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.891082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.891064 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.902731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.902494 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2"] Nov 01 22:35:59.908175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.903339 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.908175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.905780 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8rx\" (UniqueName: \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-kube-api-access-vp8rx\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:35:59.911743 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:35:59.911706 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6ba517_374b_45a0_b36c_43d08dc7fffa.slice/crio-c90a972343118bafd692145f3e94a4f0b67ae36ab548c79151d9375cc7660a92 WatchSource:0}: Error finding container c90a972343118bafd692145f3e94a4f0b67ae36ab548c79151d9375cc7660a92: Status 404 returned error can't find the container with id c90a972343118bafd692145f3e94a4f0b67ae36ab548c79151d9375cc7660a92 Nov 01 22:35:59.916245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.915857 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:59.916245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.915948 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:59.916245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.915986 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:35:59.990772 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.990733 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.990932 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.990786 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.990932 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.990830 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.990932 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.990867 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29n8t\" (UniqueName: \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-kube-api-access-29n8t\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.990932 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.990913 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.991204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.990963 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8dc27f4-6212-4492-a1d5-dcced032e3af-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.991204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991013 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.991204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991058 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.991204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991129 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.991204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991175 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.991586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991241 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.991586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991239 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.991586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991270 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.991586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991330 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.991586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991381 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.991586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991413 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xtjl\" (UniqueName: \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-kube-api-access-6xtjl\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.991586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991474 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.991586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991510 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.991586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991572 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.992050 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991924 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8dc27f4-6212-4492-a1d5-dcced032e3af-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.992132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.992035 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.992515 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.992238 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.992657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.992555 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.992657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.992555 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.992792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.992682 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.992792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.991518 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.992901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.992822 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.992901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.992829 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.993547 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.993348 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:35:59.994958 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.994932 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.995077 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.995029 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:35:59.995370 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:35:59.995353 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:36:00.001756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.001728 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:36:00.001858 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.001735 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:36:00.001902 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.001846 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29n8t\" (UniqueName: \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-kube-api-access-29n8t\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:36:00.001902 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.001853 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xtjl\" (UniqueName: \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-kube-api-access-6xtjl\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:36:00.064220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.064185 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:36:00.133614 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.132415 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:36:00.169640 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.169383 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:36:00.362119 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.362073 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk"] Nov 01 22:36:00.370043 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.370019 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:00.377003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.375100 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck"] Nov 01 22:36:00.380755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.380722 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk"] Nov 01 22:36:00.382907 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.382860 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:00.383028 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.382942 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:00.383028 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.382990 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:00.385361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.385325 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0500c01f\"" Nov 01 22:36:00.403660 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.403622 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974t5\" (UniqueName: \"kubernetes.io/projected/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-kube-api-access-974t5\") pod \"payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk\" (UID: \"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:00.404131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.404097 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk\" (UID: \"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:00.432991 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.432945 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c"] Nov 01 22:36:00.462395 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.462362 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c"] Nov 01 22:36:00.462739 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.462669 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:00.472457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.472431 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0500c01f-dockercfg-wx2pj\"" Nov 01 22:36:00.503027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.502643 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt"] Nov 01 22:36:00.504789 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:36:00.504738 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbec9fb_83d1_405b_bbcd_bc0f30257644.slice/crio-19078a17652599509c5b937e7e1a504deb708fb5a4fe4db98ede9dd04b4f1a72 WatchSource:0}: Error finding container 19078a17652599509c5b937e7e1a504deb708fb5a4fe4db98ede9dd04b4f1a72: Status 404 returned error can't find the container with id 19078a17652599509c5b937e7e1a504deb708fb5a4fe4db98ede9dd04b4f1a72 Nov 01 22:36:00.513962 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.505818 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371e5a7f-4d04-438b-8dd8-3f2d608954ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c\" (UID: \"371e5a7f-4d04-438b-8dd8-3f2d608954ed\") " pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:00.513962 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.505870 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974t5\" (UniqueName: \"kubernetes.io/projected/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-kube-api-access-974t5\") pod \"payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk\" (UID: \"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:00.513962 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.506018 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk\" (UID: \"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:00.513962 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.506043 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz869\" (UniqueName: \"kubernetes.io/projected/371e5a7f-4d04-438b-8dd8-3f2d608954ed-kube-api-access-bz869\") pod \"payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c\" (UID: \"371e5a7f-4d04-438b-8dd8-3f2d608954ed\") " pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:00.513962 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.507208 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk\" (UID: \"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:00.532628 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.532281 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:00.532628 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.532365 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:00.532628 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.532407 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:00.560162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.560025 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974t5\" (UniqueName: \"kubernetes.io/projected/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-kube-api-access-974t5\") pod \"payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk\" (UID: \"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:00.584172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.584081 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt"] Nov 01 22:36:00.593394 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:36:00.593357 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dc27f4_6212_4492_a1d5_dcced032e3af.slice/crio-2d7862b6d60f377b0378a08ed3815472caa988c6ee4c38b58c02ed094fc57115 WatchSource:0}: Error finding container 2d7862b6d60f377b0378a08ed3815472caa988c6ee4c38b58c02ed094fc57115: Status 404 returned error can't find the container with id 2d7862b6d60f377b0378a08ed3815472caa988c6ee4c38b58c02ed094fc57115 Nov 01 22:36:00.603092 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.599142 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:00.603092 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.599235 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:00.603092 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.599278 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:00.631716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.607827 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371e5a7f-4d04-438b-8dd8-3f2d608954ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c\" (UID: \"371e5a7f-4d04-438b-8dd8-3f2d608954ed\") " pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:00.631716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.608306 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz869\" (UniqueName: \"kubernetes.io/projected/371e5a7f-4d04-438b-8dd8-3f2d608954ed-kube-api-access-bz869\") pod \"payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c\" (UID: \"371e5a7f-4d04-438b-8dd8-3f2d608954ed\") " pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:00.631716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.609625 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371e5a7f-4d04-438b-8dd8-3f2d608954ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c\" (UID: \"371e5a7f-4d04-438b-8dd8-3f2d608954ed\") " pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:00.740128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.734513 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz869\" (UniqueName: \"kubernetes.io/projected/371e5a7f-4d04-438b-8dd8-3f2d608954ed-kube-api-access-bz869\") pod \"payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c\" (UID: \"371e5a7f-4d04-438b-8dd8-3f2d608954ed\") " pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:00.786695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.785278 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" event={"ID":"ec6ba517-374b-45a0-b36c-43d08dc7fffa","Type":"ContainerStarted","Data":"1bee0ba21cb23c9694c8aa7605f2dac71635252144b5ab5872ab011e4a5c6739"} Nov 01 22:36:00.786695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.785347 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" event={"ID":"ec6ba517-374b-45a0-b36c-43d08dc7fffa","Type":"ContainerStarted","Data":"c90a972343118bafd692145f3e94a4f0b67ae36ab548c79151d9375cc7660a92"} Nov 01 22:36:00.786695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.786461 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:00.829382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.813393 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:00.871178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.869028 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" event={"ID":"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2","Type":"ContainerStarted","Data":"3187ed6473fad3e6d71b6fcbf35ad085e783d2403da23bbaaf06cc95cc0915df"} Nov 01 22:36:00.871178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.869125 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" event={"ID":"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2","Type":"ContainerStarted","Data":"9a89e80a4949f6dd0b5946cf8a28f52c632bcbbe9b23e87722eef942f03a4664"} Nov 01 22:36:00.908198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.872840 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podStartSLOduration=1.872819902 podStartE2EDuration="1.872819902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:00.871079655 +0000 UTC m=+764.655518575" watchObservedRunningTime="2026-11-01 22:36:00.872819902 +0000 UTC m=+764.657258826" Nov 01 22:36:00.922014 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.915493 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" event={"ID":"b8dc27f4-6212-4492-a1d5-dcced032e3af","Type":"ContainerStarted","Data":"2d7862b6d60f377b0378a08ed3815472caa988c6ee4c38b58c02ed094fc57115"} Nov 01 22:36:00.976906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:00.976863 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" event={"ID":"fdbec9fb-83d1-405b-bbcd-bc0f30257644","Type":"ContainerStarted","Data":"19078a17652599509c5b937e7e1a504deb708fb5a4fe4db98ede9dd04b4f1a72"} Nov 01 22:36:01.099807 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:01.099774 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:36:01.123771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:01.123738 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:01.124034 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:01.123999 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:01.626689 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:01.626666 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:36:01.633251 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:01.633216 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:01.633515 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:01.633490 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:02.033333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:02.032235 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:02.130140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:02.120258 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:02.134163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:02.130413 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:02.149223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:02.140934 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:02.373465 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:02.228518 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:02.373465 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:02.228736 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:02.582563 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:02.492204 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podStartSLOduration=3.492183815 podStartE2EDuration="3.492183815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:00.943463333 +0000 UTC m=+764.727902303" watchObservedRunningTime="2026-11-01 22:36:02.492183815 +0000 UTC m=+766.276622737" Nov 01 22:36:02.612143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:02.597221 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:02.612143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:02.597282 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:02.846577 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:02.846543 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk"] Nov 01 22:36:03.683424 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:03.683319 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:03.746156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:03.713329 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:36:03.785333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:03.720367 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:04.077413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:03.767453 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:04.077413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.052200 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:04.077413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:03.848927 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:04.077413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.074821 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:36:04.077413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.074930 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:36:04.082642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:03.848726 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:04.150163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.104896 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:36:04.379286 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.111779 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:04.379583 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.379555 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:04.379728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.312223 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" podUID="7006c77f-be7e-43be-b47d-7aaef13e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:04.680268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.655291 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:04.680268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.655375 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:04.779448 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.763008 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lxpj5" podUID="ce3f4a45-aa4a-465f-a436-9c0f4a542380" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 01 22:36:04.865140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.823451 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" podUID="ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:05.062269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:04.944766 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" podUID="58de7f57-d10e-42ff-ab3e-3952b601bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:05.137542 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:05.137506 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:05.137856 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:05.137813 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:05.171135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:05.162216 6466 patch_prober.go:29] interesting pod/metrics-server-796bf69784-66sjq container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded" start-of-body= Nov 01 22:36:05.171135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:05.162298 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-796bf69784-66sjq" podUID="18595f28-dc19-4df0-a61b-905e22653608" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded" Nov 01 22:36:05.226141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:05.219102 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:05.303468 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:05.303421 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:05.303854 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:05.303820 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:05.304155 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:05.304099 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:05.355372 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:05.355271 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:05.355641 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:05.355610 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:05.605484 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:05.605379 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" podUID="7006c77f-be7e-43be-b47d-7aaef13e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:06.222199 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.221518 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Nov 01 22:36:06.303134 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:06.252317 6466 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.9s" Nov 01 22:36:06.303134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.252363 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" event={"ID":"fdbec9fb-83d1-405b-bbcd-bc0f30257644","Type":"ContainerStarted","Data":"cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679"} Nov 01 22:36:06.303134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.252417 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr"] Nov 01 22:36:06.332134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.331436 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:06.332134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.331499 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:06.496555 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.496465 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:06.753458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.753341 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:06.753698 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.548549 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 01 22:36:06.753862 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.549047 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:06.754358 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.464828 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:06.755171 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.701681 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:06.833132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.832912 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:06.833132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.756087 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:36:06.833391 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.833088 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" Nov 01 22:36:06.833391 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.833224 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr"] Nov 01 22:36:06.833391 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.833306 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:36:06.994133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.756070 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:06.994133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.714747 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:06.994133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.993958 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:06.994133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.549079 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:07.019194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.832611 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" podUID="58de7f57-d10e-42ff-ab3e-3952b601bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:07.019194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:06.549098 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:07.043130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.024736 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podStartSLOduration=8.024714087 podStartE2EDuration="8.024714087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:06.43752833 +0000 UTC m=+770.221967254" watchObservedRunningTime="2026-11-01 22:36:07.024714087 +0000 UTC m=+770.809153007" Nov 01 22:36:07.043130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.025335 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fd214936-openshift-default-dockercfg-8zfkb\"" Nov 01 22:36:07.077133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.073597 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:07.077133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.073650 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:07.249958 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.157966 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:07.249958 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.158020 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:07.367389 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.367335 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Nov 01 22:36:07.582136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.561592 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" event={"ID":"b8dc27f4-6212-4492-a1d5-dcced032e3af","Type":"ContainerStarted","Data":"63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee"} Nov 01 22:36:07.582136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.561669 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c"] Nov 01 22:36:07.601744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.601704 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:07.602013 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.601974 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:07.651158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.639454 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5c5d0c-3959-4b87-879e-b581dc09a168-istiod-ca-cert\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.651158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.639539 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-data\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.651158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.639636 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-socket\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.651158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.639840 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-credential-socket\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.651158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.639945 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-token\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.651158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.639974 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-podinfo\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.651158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.640131 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27zlq\" (UniqueName: \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-kube-api-access-27zlq\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.651158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.640265 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-envoy\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.708187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.687559 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-certs\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.810141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.808682 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" event={"ID":"371e5a7f-4d04-438b-8dd8-3f2d608954ed","Type":"ContainerStarted","Data":"509b1d117aa7779d09155ce50f0672e7ca8745a40e4efc13064250d80a592b36"} Nov 01 22:36:07.810141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.809414 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:07.810141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.809752 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:07.870070 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.870027 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-token\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.870327 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.870304 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-podinfo\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.870559 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.870542 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27zlq\" (UniqueName: \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-kube-api-access-27zlq\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.870747 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.870729 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-envoy\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.870920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.870903 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-certs\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.905181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.898498 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-token\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.909949 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.909917 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-certs\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.910541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.910508 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5c5d0c-3959-4b87-879e-b581dc09a168-istiod-ca-cert\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.920979 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.920745 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5c5d0c-3959-4b87-879e-b581dc09a168-istiod-ca-cert\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.931626 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.931588 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-data\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.932557 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.921238 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-data\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.933607 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.933064 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-socket\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.941892 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.941858 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-credential-socket\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.945802 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.945736 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-socket\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.978134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.964783 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-envoy\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.983216 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.983181 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-podinfo\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.983708 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.983446 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-credential-socket\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:07.997132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:07.994220 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.007143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.004898 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27zlq\" (UniqueName: \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-kube-api-access-27zlq\") pod \"e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:08.027263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.027207 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.090589 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.080167 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" event={"ID":"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5","Type":"ContainerStarted","Data":"bc76f16126f5e255f12001c82f314b47cafbf1c8fe32beccce5738a73b7187e0"} Nov 01 22:36:08.090589 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.080226 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" event={"ID":"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5","Type":"ContainerStarted","Data":"96e413a0748ce2e1ede4bcc5724a417a595af8541d1798ffff3e3eff76f5f7ca"} Nov 01 22:36:08.090589 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.080913 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:08.142342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.126265 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:08.142342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.126329 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:08.156406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.156283 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:08.156715 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.156661 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:08.172435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.172404 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:36:08.186130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.182723 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:08.186130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.182780 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:08.198582 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.198545 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:08.298677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.298604 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podStartSLOduration=9.298580861 podStartE2EDuration="9.298580861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:07.906953317 +0000 UTC m=+771.691392241" watchObservedRunningTime="2026-11-01 22:36:08.298580861 +0000 UTC m=+772.083019773" Nov 01 22:36:08.381765 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.381233 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" podStartSLOduration=8.381209064 podStartE2EDuration="8.381209064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:08.152318681 +0000 UTC m=+771.936757606" watchObservedRunningTime="2026-11-01 22:36:08.381209064 +0000 UTC m=+772.165647985" Nov 01 22:36:08.406283 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.406248 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:08.406585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.406524 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.650640 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.650583 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:08.650956 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.650911 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.658408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.653977 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:08.658408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.654040 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:08.730346 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.730303 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:08.730620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.730588 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.817140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.812328 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.817140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:08.816281 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:09.004687 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.004589 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:09.079136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.028811 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:09.103133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.088932 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:09.103133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.088991 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:09.168311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.168272 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:09.168586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.168553 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:09.170379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.170354 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:09.170534 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.170513 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:09.190508 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.190471 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:09.425052 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.425011 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" event={"ID":"371e5a7f-4d04-438b-8dd8-3f2d608954ed","Type":"ContainerStarted","Data":"bbc5e06f45861d270571f72bbbe03e599b067f291dd626644d40698620454687"} Nov 01 22:36:09.426946 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.426922 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:09.550135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.528151 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" podStartSLOduration=9.528130479 podStartE2EDuration="9.528130479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:09.501492297 +0000 UTC m=+773.285931218" watchObservedRunningTime="2026-11-01 22:36:09.528130479 +0000 UTC m=+773.312569398" Nov 01 22:36:09.632134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.631487 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:09.632134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.631546 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:09.832190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.827859 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:09.860144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:09.837156 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:10.100254 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:10.014555 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:10.110138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:10.109958 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:10.178134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:10.170617 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:10.178134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:10.170684 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:10.256133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:10.246299 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:10.256133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:10.246366 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:10.646306 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:10.646265 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:10.658302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:10.658260 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:10.763853 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:10.763729 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:10.771239 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:10.771186 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:10.909134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:10.880304 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:11.158988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.017323 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:36:11.158988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.060195 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:11.158988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.065753 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:11.158988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.138906 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:11.158988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.113421 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:11.172373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.172338 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:11.173402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.072307 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:11.173635 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.173610 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:11.191529 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.191491 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:11.191777 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.191748 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:11.234185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.234151 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:11.273937 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.273893 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:11.414252 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.414160 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr"] Nov 01 22:36:11.600225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.593533 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:11.600225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.593591 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:11.622407 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.601392 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:11.622407 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.601494 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:11.622407 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.601546 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:11.957185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:11.953057 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:12.202131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:12.202001 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:12.202335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:12.202309 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:12.205214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:12.205179 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:12.467128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:12.464508 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:12.467128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:12.464563 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:12.660156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:12.651561 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:12.686138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:12.660450 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:12.703246 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:12.464468 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:12.703588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:12.703544 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:13.032140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.029394 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:13.106142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.086726 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:13.106142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.086803 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:13.106142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.098158 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:13.106142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.098212 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:13.135088 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.135051 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:13.135394 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.135367 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:13.246133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.243203 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:13.286193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.246655 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:13.562353 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.554662 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:13.562353 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.292288 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:13.641132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.292227 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:13.796133 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:13.644137 6466 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.296s" Nov 01 22:36:13.796133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.764643 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:36:13.796133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.764722 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:36:13.796133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.764742 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" event={"ID":"9d5c5d0c-3959-4b87-879e-b581dc09a168","Type":"ContainerStarted","Data":"a939c1ca9e23fedf0ff5dd7170cd1957512dab1073b66a69c990e3b291cc4688"} Nov 01 22:36:13.798162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.730851 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:13.798162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.797531 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:13.841185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:13.837998 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:36:14.086242 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:14.067199 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:14.086242 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:14.067270 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:14.161361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:14.161321 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:14.169669 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:14.161588 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:14.439190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:14.424612 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:14.439486 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:14.439452 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:14.605132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:14.605045 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:14.646134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:14.627964 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:14.646134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:14.628023 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:14.802277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:14.789376 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:14.957133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:14.951281 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:15.012135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.011342 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:36:15.028164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.023246 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:15.072285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.072195 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:36:15.074013 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.073986 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:15.074287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.074251 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:15.136537 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.136498 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:15.137321 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.137286 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:15.137956 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.137932 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:36:15.172247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.172215 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:36:15.187007 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.186973 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:15.204975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.204930 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:15.603483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.595284 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:15.603483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.595363 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:15.717168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.697502 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" event={"ID":"9d5c5d0c-3959-4b87-879e-b581dc09a168","Type":"ContainerStarted","Data":"184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867"} Nov 01 22:36:15.717168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.698897 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:15.772605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.772541 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" podStartSLOduration=12.772518664 podStartE2EDuration="12.772518664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:15.743136563 +0000 UTC m=+779.527575485" watchObservedRunningTime="2026-11-01 22:36:15.772518664 +0000 UTC m=+779.556957586" Nov 01 22:36:15.795335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.795295 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:15.954270 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.954222 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:15.956086 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.956061 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:36:15.974134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:15.961520 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:16.037724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.037686 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:16.071343 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.065099 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:16.071651 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.071605 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:16.073479 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.072417 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:36:16.135291 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.133602 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:16.135291 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.133657 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:16.188279 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.188229 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:16.225273 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.225103 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:16.228596 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.228564 6466 patch_prober.go:29] interesting pod/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:36:16.228873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.228831 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" podUID="9d5c5d0c-3959-4b87-879e-b581dc09a168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 01 22:36:16.255176 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.228722 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:16.255432 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.255400 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:16.590884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.590790 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:16.591152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.591096 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:16.726362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.705773 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:16.819134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.802792 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:16.822615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.822579 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:36:16.944371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:16.944330 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:17.039135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.009673 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:17.115352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.115322 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:36:17.157317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.157269 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:17.157617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.157560 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:17.347329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.305796 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:17.347329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.331903 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:17.347329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.331958 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:17.347329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.332723 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:17.347329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.332767 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:17.595893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.595862 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:17.630610 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.603292 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:17.940135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:17.939214 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:18.043896 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:18.043745 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:18.073133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:18.069224 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:18.073133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:18.071199 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:18.073133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:18.071245 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:18.140148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:18.138444 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:18.140148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:18.138503 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:18.215140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:18.215077 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:18.215324 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:18.215243 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:18.611187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:18.611150 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:18.654362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:18.638226 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:18.910195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:18.843075 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:19.089604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.070621 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:19.089604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.071664 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:19.089604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.075611 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:19.159327 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.159280 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:19.176968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.176888 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:19.202229 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.189147 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:19.202543 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.177345 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:19.202692 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.202655 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:19.512220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.508055 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q"] Nov 01 22:36:19.579506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.579457 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:19.579742 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.579718 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:19.580288 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.580260 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:19.600242 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:19.600190 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:19.864892 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:19.864797 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:36:20.058180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.058142 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q"] Nov 01 22:36:20.137480 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.137459 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:20.138067 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.138049 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:36:20.138309 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.138276 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:36:20.161931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.088375 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:20.162428 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.088548 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:20.162624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.162599 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:20.162754 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.076262 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:20.174170 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.135236 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:36:20.214158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.192841 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:20.214158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.192900 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:20.214158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.146008 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:20.214158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.211244 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:20.214158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.076209 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:20.214158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.211665 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:36:20.214158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.088334 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:20.348175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.269014 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" podUID="ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:20.348175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.269126 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" podUID="20f8556e-bad5-456f-ad1a-14ef9a2ea949" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 01 22:36:20.348175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.285539 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fd214936\"" Nov 01 22:36:20.398301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.373630 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e22b11-3488-4ba5-b4cb-306be966e90d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q\" (UID: \"d2e22b11-3488-4ba5-b4cb-306be966e90d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:36:20.398301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.373764 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnlc4\" (UniqueName: \"kubernetes.io/projected/d2e22b11-3488-4ba5-b4cb-306be966e90d-kube-api-access-hnlc4\") pod \"payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q\" (UID: \"d2e22b11-3488-4ba5-b4cb-306be966e90d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:36:20.427654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.420335 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:36:20.704191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.675254 6466 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:36:20.704191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.675315 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:36:20.704191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.677731 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnlc4\" (UniqueName: \"kubernetes.io/projected/d2e22b11-3488-4ba5-b4cb-306be966e90d-kube-api-access-hnlc4\") pod \"payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q\" (UID: \"d2e22b11-3488-4ba5-b4cb-306be966e90d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:36:20.704191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.678195 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e22b11-3488-4ba5-b4cb-306be966e90d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q\" (UID: \"d2e22b11-3488-4ba5-b4cb-306be966e90d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:36:20.893128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.825234 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnlc4\" (UniqueName: \"kubernetes.io/projected/d2e22b11-3488-4ba5-b4cb-306be966e90d-kube-api-access-hnlc4\") pod \"payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q\" (UID: \"d2e22b11-3488-4ba5-b4cb-306be966e90d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:36:20.932338 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.932203 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e22b11-3488-4ba5-b4cb-306be966e90d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q\" (UID: \"d2e22b11-3488-4ba5-b4cb-306be966e90d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:36:20.969437 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:20.969344 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:21.017472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.017428 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:36:21.073297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.065423 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:21.073297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.065480 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:21.210141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.191284 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:21.210141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.191348 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:21.210141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.191962 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:21.210141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.192004 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:21.210141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.193957 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:36:21.212388 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.211389 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:21.230778 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.230467 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:21.303138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.301440 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:36:21.372143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.365733 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:36:21.830546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:21.791613 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q"] Nov 01 22:36:22.065035 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.065011 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:36:22.065547 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.065520 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 22:36:22.133376 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.133338 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:36:22.133607 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.133396 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 01 22:36:22.173152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.170325 6466 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:36:22.173152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.170393 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 22:36:22.300782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.300682 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:36:22.302897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.302869 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:36:22.463908 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.463869 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" event={"ID":"d2e22b11-3488-4ba5-b4cb-306be966e90d","Type":"ContainerStarted","Data":"000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d"} Nov 01 22:36:22.464151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.463963 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" event={"ID":"d2e22b11-3488-4ba5-b4cb-306be966e90d","Type":"ContainerStarted","Data":"a1a8da461d1a7118590d07f8b464a7064b15b9599a8ced48864c9ebd19740c61"} Nov 01 22:36:22.464151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.464016 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:36:22.489417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:22.489364 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" podStartSLOduration=4.489350048 podStartE2EDuration="4.489350048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:22.48359881 +0000 UTC m=+786.268037732" watchObservedRunningTime="2026-11-01 22:36:22.489350048 +0000 UTC m=+786.273788968" Nov 01 22:36:23.490202 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:23.489021 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:36:23.494241 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:23.494207 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:36:23.538446 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:23.538415 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:36:23.548529 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:23.548502 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:36:23.700158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:23.700128 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:36:23.701455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:23.701433 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:36:24.968018 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:24.967989 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9"] Nov 01 22:36:24.979974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:24.979947 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.007400 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.007377 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fd214936-serving-cert\"" Nov 01 22:36:25.020589 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.020557 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9"] Nov 01 22:36:25.045882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.045848 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3990f82b-1cdb-47d7-986f-2a20f096f3b6-openshift-service-ca\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.046063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.046009 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-metrics-certs\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.046205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.046185 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-maas-api-tls\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.046274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.046222 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7sd9\" (UniqueName: \"kubernetes.io/projected/3990f82b-1cdb-47d7-986f-2a20f096f3b6-kube-api-access-x7sd9\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.179133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.170861 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3990f82b-1cdb-47d7-986f-2a20f096f3b6-openshift-service-ca\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.202329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.202288 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-metrics-certs\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.202793 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.202765 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-maas-api-tls\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.202979 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.202957 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7sd9\" (UniqueName: \"kubernetes.io/projected/3990f82b-1cdb-47d7-986f-2a20f096f3b6-kube-api-access-x7sd9\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.203891 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.171984 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3990f82b-1cdb-47d7-986f-2a20f096f3b6-openshift-service-ca\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.218183 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:25.208820 6466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-serving-cert: secret "maas-api-e2e-ait-fd214936-serving-cert" not found Nov 01 22:36:25.218183 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:25.216332 6466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-maas-api-tls podName:3990f82b-1cdb-47d7-986f-2a20f096f3b6 nodeName:}" failed. No retries permitted until 2026-11-01 22:36:25.716281528 +0000 UTC m=+789.500720443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-maas-api-tls") pod "maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" (UID: "3990f82b-1cdb-47d7-986f-2a20f096f3b6") : secret "maas-api-e2e-ait-fd214936-serving-cert" not found Nov 01 22:36:25.291786 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.291751 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-metrics-certs\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.292063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.292032 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7sd9\" (UniqueName: \"kubernetes.io/projected/3990f82b-1cdb-47d7-986f-2a20f096f3b6-kube-api-access-x7sd9\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.718846 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.718807 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-maas-api-tls\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.785925 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.785874 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-maas-api-tls\") pod \"maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:25.895455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:25.895417 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:26.251039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:26.241779 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9"] Nov 01 22:36:26.350914 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:26.350887 6466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:36:26.532079 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:26.531991 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" event={"ID":"3990f82b-1cdb-47d7-986f-2a20f096f3b6","Type":"ContainerStarted","Data":"f0916332b49057d2e0ea2c8e8a82e5a81059428dca786fcc3d164b057c4b6819"} Nov 01 22:36:27.541441 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:27.541397 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" event={"ID":"3990f82b-1cdb-47d7-986f-2a20f096f3b6","Type":"ContainerStarted","Data":"7464c39bb32678e1b8ecdf076018432f5c88dd528a36906e65dc1f7af321148b"} Nov 01 22:36:27.541854 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:27.541483 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:27.564045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:27.563986 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" podStartSLOduration=3.101765744 podStartE2EDuration="3.563968369s" totalImagesPullingTime="462.202625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:27.55945267 +0000 UTC m=+791.343891591" watchObservedRunningTime="2026-11-01 22:36:27.563968369 +0000 UTC m=+791.348407291" Nov 01 22:36:29.705360 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.705324 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp"] Nov 01 22:36:29.710082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.710065 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.722762 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.722738 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp"] Nov 01 22:36:29.828304 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.828255 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwvb\" (UniqueName: \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-kube-api-access-9rwvb\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.828473 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.828308 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.828473 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.828414 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2417aba7-f7aa-49df-900b-659dfdfff057-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.828610 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.828564 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.828695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.828672 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.828774 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.828747 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.828908 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.828889 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.828983 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.828923 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.829155 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.829125 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2417aba7-f7aa-49df-900b-659dfdfff057-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.930642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.930609 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.930642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.930646 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.930847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.930736 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2417aba7-f7aa-49df-900b-659dfdfff057-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.930847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.930800 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwvb\" (UniqueName: \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-kube-api-access-9rwvb\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.930847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.930817 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.930847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.930841 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2417aba7-f7aa-49df-900b-659dfdfff057-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.931018 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.930920 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.931018 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.930992 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.931331 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.931052 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.931331 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.931103 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.931331 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.931249 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.931510 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.931408 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.931510 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.931491 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.931712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.931686 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2417aba7-f7aa-49df-900b-659dfdfff057-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.933154 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.933134 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.933341 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.933324 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2417aba7-f7aa-49df-900b-659dfdfff057-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:29.995241 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:29.994694 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwvb\" (UniqueName: \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-kube-api-access-9rwvb\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:30.014326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:30.011321 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:30.034450 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:30.034028 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:30.380915 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:36:30.380884 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2417aba7_f7aa_49df_900b_659dfdfff057.slice/crio-ed5f8faa0c3cf46138311e01172319dd9bfd55143d9c6c7d485e753a278c84ca WatchSource:0}: Error finding container ed5f8faa0c3cf46138311e01172319dd9bfd55143d9c6c7d485e753a278c84ca: Status 404 returned error can't find the container with id ed5f8faa0c3cf46138311e01172319dd9bfd55143d9c6c7d485e753a278c84ca Nov 01 22:36:30.381097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:30.381076 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp"] Nov 01 22:36:30.383524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:30.383491 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:30.383608 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:30.383558 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:30.383608 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:30.383588 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:36:30.566161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:30.566120 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" event={"ID":"2417aba7-f7aa-49df-900b-659dfdfff057","Type":"ContainerStarted","Data":"8ddb9378ff27bd1d913ca0a60b56480579290d1e5a94d2293e76b513ba526037"} Nov 01 22:36:30.566334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:30.566183 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" event={"ID":"2417aba7-f7aa-49df-900b-659dfdfff057","Type":"ContainerStarted","Data":"ed5f8faa0c3cf46138311e01172319dd9bfd55143d9c6c7d485e753a278c84ca"} Nov 01 22:36:30.634907 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:30.634803 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podStartSLOduration=1.6347893999999998 podStartE2EDuration="1.6347894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:30.633824008 +0000 UTC m=+794.418262942" watchObservedRunningTime="2026-11-01 22:36:30.6347894 +0000 UTC m=+794.419228320" Nov 01 22:36:31.049119 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:31.048723 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:31.054579 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:31.054533 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:36:31.054854 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:31.054822 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 22:36:32.101399 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:32.101357 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:36:32.101901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:32.101419 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 22:36:33.041134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:33.035337 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:36:33.041134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:33.035408 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 22:36:33.513416 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:33.513384 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:36:33.573926 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:33.573899 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:36:34.034845 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:34.034813 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:36:34.035000 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:34.034882 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 22:36:35.050132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:35.035175 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:36:35.050132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:35.035243 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 22:36:36.037610 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:36.037568 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:36:36.037922 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:36.037890 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 22:36:37.034966 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:37.034931 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:36:37.035420 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:37.034983 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 22:36:38.037889 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:38.037847 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:36:38.042231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:38.042182 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 22:36:39.033713 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:39.033672 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:39.182692 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:39.050208 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.196189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:39.126806 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.196684 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:39.166743 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:36:39.197073 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:39.197029 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 22:36:39.475293 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:39.250347 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.487207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:39.288259 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:39.487207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:39.485642 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.522132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:39.487486 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:40.081692 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:40.078417 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:36:40.081692 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:40.078482 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 22:36:40.180931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:40.180885 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:40.197357 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:40.188210 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:36:40.229132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:40.220596 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:40.229132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:40.224581 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:36:40.502269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:40.502222 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:40.545959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:40.545885 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:40.855189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:40.808727 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:36:41.075134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:41.059236 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:36:41.092339 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:41.092303 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:36:41.092809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:41.092683 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 22:36:41.201168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:41.188211 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:41.225706 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:41.225654 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:41.569744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:41.560801 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c"] Nov 01 22:36:41.667135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:41.611336 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:41.667135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:41.613671 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.190132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:42.189383 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.269428 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:42.269379 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:42.620091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:42.619983 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.625147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:42.621853 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:42.876192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:42.853052 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:42.894071 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:42.894030 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.979143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:42.971844 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:42.979143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:42.971966 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:43.221131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:43.208180 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:43.344806 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:43.344755 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:43.534201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:43.530541 6466 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:43.534201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:43.530619 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:43.534201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:43.531881 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:43.534201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:43.531925 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:43.747134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:43.743575 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:43.882697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:43.882589 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:44.126832 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:44.126795 6466 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.779s" Nov 01 22:36:44.150403 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:44.150378 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk"] Nov 01 22:36:44.390823 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:44.390770 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.612658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:44.612552 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.637744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:44.625218 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:44.637744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:44.625296 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.789193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:44.767284 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.931351 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:44.931299 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:45.085135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.082798 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:45.187197 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.186458 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-9vghr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:45.187197 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.186533 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.315231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.304203 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:45.315231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.304270 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.399134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.398663 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.621074 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.620963 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.711647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.711595 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:45.730277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.730229 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.799134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.798202 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:45.938803 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.938759 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:45.997812 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:45.997759 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.185966 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.185924 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.383304 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.383209 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:46.383557 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.383527 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.445506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.445463 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:46.482324 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.482288 6466 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-whtxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:46.482604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.482568 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.600135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.585594 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" podUID="371e5a7f-4d04-438b-8dd8-3f2d608954ed" containerName="payload-processing" containerID="cri-o://bbc5e06f45861d270571f72bbbe03e599b067f291dd626644d40698620454687" gracePeriod=25 Nov 01 22:36:46.623276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.621744 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.800131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.798237 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:46.800131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.798305 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.800131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.798972 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.948365 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:46.937527 6466 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371e5a7f_4d04_438b_8dd8_3f2d608954ed.slice/crio-bbc5e06f45861d270571f72bbbe03e599b067f291dd626644d40698620454687.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:46.948706 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.948657 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.949628 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:46.942824 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.965168 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:46.941530 6466 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371e5a7f_4d04_438b_8dd8_3f2d608954ed.slice/crio-bbc5e06f45861d270571f72bbbe03e599b067f291dd626644d40698620454687.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371e5a7f_4d04_438b_8dd8_3f2d608954ed.slice/crio-conmon-bbc5e06f45861d270571f72bbbe03e599b067f291dd626644d40698620454687.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:47.498136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:47.497181 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:47.568131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:47.563691 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:47.603286 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:47.594447 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:36:47.622565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:47.622507 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:47.828202 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:47.801874 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:47.951422 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:47.951380 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:48.566132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:48.564521 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:48.626140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:48.624936 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:48.889001 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:48.883857 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:48.953615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:48.953561 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:49.116999 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:49.116878 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:49.241958 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:49.241839 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" podUID="d2ad4449-c9e5-431f-9f17-c3aec7bba2e5" containerName="payload-pre-processing" containerID="cri-o://bc76f16126f5e255f12001c82f314b47cafbf1c8fe32beccce5738a73b7187e0" gracePeriod=25 Nov 01 22:36:49.262157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:49.258802 6466 generic.go:356] "Generic (PLEG): container finished" podID="371e5a7f-4d04-438b-8dd8-3f2d608954ed" containerID="bbc5e06f45861d270571f72bbbe03e599b067f291dd626644d40698620454687" exitCode=0 Nov 01 22:36:49.575142 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:49.571607 6466 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.232s" Nov 01 22:36:49.651135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:49.637235 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:49.662177 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:49.662144 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" event={"ID":"371e5a7f-4d04-438b-8dd8-3f2d608954ed","Type":"ContainerDied","Data":"bbc5e06f45861d270571f72bbbe03e599b067f291dd626644d40698620454687"} Nov 01 22:36:49.662636 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:49.662082 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:49.897037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:49.896993 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:49.980967 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:49.980921 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:36:50.817157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:50.721397 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:50.817157 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:50.721748 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:36:50.817157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:50.786945 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:50.906339 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:50.902412 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:51.021133 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:36:51.021052 6466 logging.go:55] [core] [Channel #451 SubChannel #452]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.82:50052", ServerName: "10.134.0.82:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.82:50052: connect: connection refused" Nov 01 22:36:51.083156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.081461 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:51.245235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.170155 6466 patch_prober.go:29] interesting pod/thanos-querier-675964fd8b-v45n9 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9091/-/ready\": context deadline exceeded" start-of-body= Nov 01 22:36:51.871385 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.871332 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-675964fd8b-v45n9" podUID="7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/ready\": context deadline exceeded" Nov 01 22:36:51.875767 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.273647 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-dcdfx" podUID="cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc" containerName="server" probeResult="failure" output="Get \"https://10.134.0.26:8443/healthz\": context deadline exceeded" Nov 01 22:36:51.875940 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.273737 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qkfnz" podUID="58de7f57-d10e-42ff-ab3e-3952b601bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:51.894218 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.604597 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5758p" podUID="20f8556e-bad5-456f-ad1a-14ef9a2ea949" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:51.894664 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.639591 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh" podUID="7006c77f-be7e-43be-b47d-7aaef13e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:51.894905 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.238150 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h" podUID="ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:51.923465 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.923427 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:51.963278 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.927268 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:51.964205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.927319 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:51.964666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.945385 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" podUID="371e5a7f-4d04-438b-8dd8-3f2d608954ed" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.83:50052\" within 1s: context deadline exceeded" Nov 01 22:36:51.964849 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:36:51.945488 6466 logging.go:55] [core] [Channel #449 SubChannel #450]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.83:50052", ServerName: "10.134.0.83:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.83:50052: operation was canceled" Nov 01 22:36:51.979079 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.973206 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:51.979079 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.973266 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:51.988357 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.984237 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:51.988357 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.984301 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:51.988357 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.984408 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:51.988357 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:51.984434 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:52.135829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.010827 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" podUID="d2ad4449-c9e5-431f-9f17-c3aec7bba2e5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.82:50052\" within 1s: context deadline exceeded" Nov 01 22:36:52.135829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.082792 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:52.135829 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:52.104292 6466 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.765s" Nov 01 22:36:52.135829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.104339 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47"] Nov 01 22:36:52.135829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.121617 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47"] Nov 01 22:36:52.135829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.122813 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:36:52.199180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.146214 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-26aaf165\"" Nov 01 22:36:52.199180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.160170 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-26aaf165-dockercfg-k97cr\"" Nov 01 22:36:52.210375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.200862 6466 generic.go:356] "Generic (PLEG): container finished" podID="d2ad4449-c9e5-431f-9f17-c3aec7bba2e5" containerID="bc76f16126f5e255f12001c82f314b47cafbf1c8fe32beccce5738a73b7187e0" exitCode=0 Nov 01 22:36:52.210375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.201079 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" event={"ID":"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5","Type":"ContainerDied","Data":"bc76f16126f5e255f12001c82f314b47cafbf1c8fe32beccce5738a73b7187e0"} Nov 01 22:36:52.300352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.300301 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fbb9\" (UniqueName: \"kubernetes.io/projected/378b04d2-a448-4652-967c-c9c69071da63-kube-api-access-5fbb9\") pod \"payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47\" (UID: \"378b04d2-a448-4652-967c-c9c69071da63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:36:52.313485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.300781 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378b04d2-a448-4652-967c-c9c69071da63-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47\" (UID: \"378b04d2-a448-4652-967c-c9c69071da63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:36:52.428132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.423491 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378b04d2-a448-4652-967c-c9c69071da63-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47\" (UID: \"378b04d2-a448-4652-967c-c9c69071da63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:36:52.428132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.423630 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fbb9\" (UniqueName: \"kubernetes.io/projected/378b04d2-a448-4652-967c-c9c69071da63-kube-api-access-5fbb9\") pod \"payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47\" (UID: \"378b04d2-a448-4652-967c-c9c69071da63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:36:52.428132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.424961 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378b04d2-a448-4652-967c-c9c69071da63-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47\" (UID: \"378b04d2-a448-4652-967c-c9c69071da63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:36:52.486144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.479827 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fbb9\" (UniqueName: \"kubernetes.io/projected/378b04d2-a448-4652-967c-c9c69071da63-kube-api-access-5fbb9\") pod \"payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47\" (UID: \"378b04d2-a448-4652-967c-c9c69071da63\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:36:52.559134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.559089 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:36:52.647994 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.647961 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:52.891154 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.891102 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371e5a7f-4d04-438b-8dd8-3f2d608954ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371e5a7f-4d04-438b-8dd8-3f2d608954ed-plugins-config-volume\") pod \"371e5a7f-4d04-438b-8dd8-3f2d608954ed\" (UID: \"371e5a7f-4d04-438b-8dd8-3f2d608954ed\") " Nov 01 22:36:52.913595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.913463 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371e5a7f-4d04-438b-8dd8-3f2d608954ed-kube-api-access-bz869\" (UniqueName: \"kubernetes.io/projected/371e5a7f-4d04-438b-8dd8-3f2d608954ed-kube-api-access-bz869\") pod \"371e5a7f-4d04-438b-8dd8-3f2d608954ed\" (UID: \"371e5a7f-4d04-438b-8dd8-3f2d608954ed\") " Nov 01 22:36:52.914069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.914042 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371e5a7f-4d04-438b-8dd8-3f2d608954ed-plugins-config-volume" pod "371e5a7f-4d04-438b-8dd8-3f2d608954ed" (UID: "371e5a7f-4d04-438b-8dd8-3f2d608954ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:52.915739 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.915707 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371e5a7f-4d04-438b-8dd8-3f2d608954ed-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:52.964116 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.964055 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:52.966518 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.964403 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:52.966518 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:52.964455 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:53.065251 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:53.065175 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371e5a7f-4d04-438b-8dd8-3f2d608954ed-kube-api-access-bz869" pod "371e5a7f-4d04-438b-8dd8-3f2d608954ed" (UID: "371e5a7f-4d04-438b-8dd8-3f2d608954ed"). InnerVolumeSpecName "kube-api-access-bz869". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:53.110132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:53.093192 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:53.197321 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:53.170036 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz869\" (UniqueName: \"kubernetes.io/projected/371e5a7f-4d04-438b-8dd8-3f2d608954ed-kube-api-access-bz869\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:54.079550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:53.973410 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:36:54.079550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:54.076027 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:54.079550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:54.077543 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:36:54.662134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:54.650297 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:54.838161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:54.835150 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" Nov 01 22:36:54.867134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:54.838579 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:36:54.867632 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:54.867600 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c" event={"ID":"371e5a7f-4d04-438b-8dd8-3f2d608954ed","Type":"ContainerDied","Data":"509b1d117aa7779d09155ce50f0672e7ca8745a40e4efc13064250d80a592b36"} Nov 01 22:36:54.867900 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:54.867886 6466 scope.go:122] "RemoveContainer" containerID="bbc5e06f45861d270571f72bbbe03e599b067f291dd626644d40698620454687" Nov 01 22:36:54.868211 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:54.849495 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:54.987141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:54.987040 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:36:54.988508 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:54.988486 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:36:54.992632 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:54.992615 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:36:55.009008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.004756 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c"] Nov 01 22:36:55.009381 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:55.009356 6466 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/systemd-tmpfiles-clean.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371e5a7f_4d04_438b_8dd8_3f2d608954ed.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371e5a7f_4d04_438b_8dd8_3f2d608954ed.slice/crio-509b1d117aa7779d09155ce50f0672e7ca8745a40e4efc13064250d80a592b36\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:55.009967 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:55.009923 6466 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/systemd-tmpfiles-clean.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371e5a7f_4d04_438b_8dd8_3f2d608954ed.slice/crio-509b1d117aa7779d09155ce50f0672e7ca8745a40e4efc13064250d80a592b36\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371e5a7f_4d04_438b_8dd8_3f2d608954ed.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:55.010572 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.010548 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0500c01f-655cbffdbf-dqc9c"] Nov 01 22:36:55.028793 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.028758 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-plugins-config-volume\") pod \"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5\" (UID: \"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5\") " Nov 01 22:36:55.028964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.028826 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-kube-api-access-974t5\" (UniqueName: \"kubernetes.io/projected/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-kube-api-access-974t5\") pod \"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5\" (UID: \"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5\") " Nov 01 22:36:55.029466 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.029439 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-plugins-config-volume" pod "d2ad4449-c9e5-431f-9f17-c3aec7bba2e5" (UID: "d2ad4449-c9e5-431f-9f17-c3aec7bba2e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:55.029978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.029951 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:55.032149 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.032100 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-kube-api-access-974t5" pod "d2ad4449-c9e5-431f-9f17-c3aec7bba2e5" (UID: "d2ad4449-c9e5-431f-9f17-c3aec7bba2e5"). InnerVolumeSpecName "kube-api-access-974t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:55.105294 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:36:55.105240 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378b04d2_a448_4652_967c_c9c69071da63.slice/crio-1afba2397004d8554a3dfe7556a3ffba655abca43520fa10cae0dcc485f68e0c WatchSource:0}: Error finding container 1afba2397004d8554a3dfe7556a3ffba655abca43520fa10cae0dcc485f68e0c: Status 404 returned error can't find the container with id 1afba2397004d8554a3dfe7556a3ffba655abca43520fa10cae0dcc485f68e0c Nov 01 22:36:55.107436 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.107406 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47"] Nov 01 22:36:55.132540 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.132390 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974t5\" (UniqueName: \"kubernetes.io/projected/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5-kube-api-access-974t5\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:55.866765 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.866726 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" event={"ID":"d2ad4449-c9e5-431f-9f17-c3aec7bba2e5","Type":"ContainerDied","Data":"96e413a0748ce2e1ede4bcc5724a417a595af8541d1798ffff3e3eff76f5f7ca"} Nov 01 22:36:55.867333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.867306 6466 scope.go:122] "RemoveContainer" containerID="bc76f16126f5e255f12001c82f314b47cafbf1c8fe32beccce5738a73b7187e0" Nov 01 22:36:55.867645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.867622 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk" Nov 01 22:36:55.895671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.894203 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" event={"ID":"378b04d2-a448-4652-967c-c9c69071da63","Type":"ContainerStarted","Data":"0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687"} Nov 01 22:36:55.895671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.894274 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" event={"ID":"378b04d2-a448-4652-967c-c9c69071da63","Type":"ContainerStarted","Data":"1afba2397004d8554a3dfe7556a3ffba655abca43520fa10cae0dcc485f68e0c"} Nov 01 22:36:55.895671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.895595 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:36:55.972815 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.972759 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" podStartSLOduration=4.972740568 podStartE2EDuration="4.972740568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:55.925742839 +0000 UTC m=+819.710181761" watchObservedRunningTime="2026-11-01 22:36:55.972740568 +0000 UTC m=+819.757179490" Nov 01 22:36:55.975995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.975967 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk"] Nov 01 22:36:55.978885 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:55.978859 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0500c01f-69f5b6bf9f-ns5kk"] Nov 01 22:36:56.368731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:56.368692 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371e5a7f-4d04-438b-8dd8-3f2d608954ed" path="/var/lib/kubelet/pods/371e5a7f-4d04-438b-8dd8-3f2d608954ed/volumes" Nov 01 22:36:56.369262 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:56.369247 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ad4449-c9e5-431f-9f17-c3aec7bba2e5" path="/var/lib/kubelet/pods/d2ad4449-c9e5-431f-9f17-c3aec7bba2e5/volumes" Nov 01 22:36:58.148401 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.138839 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8"] Nov 01 22:36:58.148401 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.139230 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" podUID="28632ab8-53e2-4244-82e1-e0daa463be0d" containerName="istio-proxy" containerID="cri-o://70112b71f1a9c8a747cbfd6ffbe4e2ede4c9cb3fa726b16693178841760fea6e" gracePeriod=30 Nov 01 22:36:58.666233 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.666189 6466 patch_prober.go:29] interesting pod/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:36:58.666444 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.666260 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" podUID="28632ab8-53e2-4244-82e1-e0daa463be0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:36:58.925037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.924958 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4"] Nov 01 22:36:58.926168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.926147 6466 cpu_manager.go:414] "containerMap: removing container" podUID="371e5a7f-4d04-438b-8dd8-3f2d608954ed" containerName="payload-processing" Nov 01 22:36:58.926265 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.926170 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371e5a7f-4d04-438b-8dd8-3f2d608954ed" containerName="payload-processing" Nov 01 22:36:58.926309 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.926265 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d2ad4449-c9e5-431f-9f17-c3aec7bba2e5" containerName="payload-pre-processing" Nov 01 22:36:58.926309 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.926273 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ad4449-c9e5-431f-9f17-c3aec7bba2e5" containerName="payload-pre-processing" Nov 01 22:36:58.926440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.926429 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="371e5a7f-4d04-438b-8dd8-3f2d608954ed" containerName="payload-processing" Nov 01 22:36:58.926475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.926457 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ad4449-c9e5-431f-9f17-c3aec7bba2e5" containerName="payload-pre-processing" Nov 01 22:36:58.932061 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.932039 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:58.940003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.939973 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-26aaf165-serving-cert\"" Nov 01 22:36:58.944957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.944929 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4"] Nov 01 22:36:58.977161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.977119 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7333c3-94fd-4589-875b-43d81c90a0be-openshift-service-ca\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:58.977562 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.977529 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks9br\" (UniqueName: \"kubernetes.io/projected/0e7333c3-94fd-4589-875b-43d81c90a0be-kube-api-access-ks9br\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:58.977680 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.977579 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-metrics-certs\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:58.977898 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:58.977870 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-maas-api-tls\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:59.083714 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:59.082465 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7333c3-94fd-4589-875b-43d81c90a0be-openshift-service-ca\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:59.083714 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:59.082616 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks9br\" (UniqueName: \"kubernetes.io/projected/0e7333c3-94fd-4589-875b-43d81c90a0be-kube-api-access-ks9br\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:59.083714 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:59.082649 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-metrics-certs\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:59.083714 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:59.082902 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-maas-api-tls\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:59.083714 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:59.083156 6466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-serving-cert: secret "maas-api-e2e-derive-26aaf165-serving-cert" not found Nov 01 22:36:59.083714 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:36:59.083229 6466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-maas-api-tls podName:0e7333c3-94fd-4589-875b-43d81c90a0be nodeName:}" failed. No retries permitted until 2026-11-01 22:36:59.58320492 +0000 UTC m=+823.367643838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-maas-api-tls") pod "maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" (UID: "0e7333c3-94fd-4589-875b-43d81c90a0be") : secret "maas-api-e2e-derive-26aaf165-serving-cert" not found Nov 01 22:36:59.088166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:59.084331 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7333c3-94fd-4589-875b-43d81c90a0be-openshift-service-ca\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:59.130140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:59.127436 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-metrics-certs\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:59.130140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:59.128825 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks9br\" (UniqueName: \"kubernetes.io/projected/0e7333c3-94fd-4589-875b-43d81c90a0be-kube-api-access-ks9br\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:59.227559 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:59.226002 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q"] Nov 01 22:36:59.642186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:59.637885 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-maas-api-tls\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:59.715133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:59.701926 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-maas-api-tls\") pod \"maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:36:59.862133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:36:59.860804 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:37:00.538375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:00.538348 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4"] Nov 01 22:37:00.539704 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:37:00.539671 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7333c3_94fd_4589_875b_43d81c90a0be.slice/crio-bf88f4e7381151a6e3fba3c2714b75f3f7adf54fedf66a080d97279fd0364ca7 WatchSource:0}: Error finding container bf88f4e7381151a6e3fba3c2714b75f3f7adf54fedf66a080d97279fd0364ca7: Status 404 returned error can't find the container with id bf88f4e7381151a6e3fba3c2714b75f3f7adf54fedf66a080d97279fd0364ca7 Nov 01 22:37:01.156031 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:01.155997 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" event={"ID":"0e7333c3-94fd-4589-875b-43d81c90a0be","Type":"ContainerStarted","Data":"70492cca2f4ae70939b23876e9e80038fe3b53d509d19dfbb864f2a9c14b4d9b"} Nov 01 22:37:01.156251 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:01.156073 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" event={"ID":"0e7333c3-94fd-4589-875b-43d81c90a0be","Type":"ContainerStarted","Data":"bf88f4e7381151a6e3fba3c2714b75f3f7adf54fedf66a080d97279fd0364ca7"} Nov 01 22:37:01.156251 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:01.156241 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:37:01.177970 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:01.177918 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" podStartSLOduration=2.67594096 podStartE2EDuration="3.177904799s" totalImagesPullingTime="501.963839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:01.176673643 +0000 UTC m=+824.961112608" watchObservedRunningTime="2026-11-01 22:37:01.177904799 +0000 UTC m=+824.962343737" Nov 01 22:37:03.371304 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.371269 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8"] Nov 01 22:37:03.376033 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.376013 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:03.389626 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.389580 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-61ce6e07\"" Nov 01 22:37:03.401046 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.401006 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68"] Nov 01 22:37:03.426611 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.416776 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:03.426611 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.419367 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8rfb\" (UniqueName: \"kubernetes.io/projected/e867a1df-a0ee-496b-a331-3d6ade14bee2-kube-api-access-q8rfb\") pod \"payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8\" (UID: \"e867a1df-a0ee-496b-a331-3d6ade14bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:03.426611 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.419599 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e867a1df-a0ee-496b-a331-3d6ade14bee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8\" (UID: \"e867a1df-a0ee-496b-a331-3d6ade14bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:03.426611 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.425383 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8"] Nov 01 22:37:03.427515 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.427475 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-61ce6e07-dockercfg-v6z5c\"" Nov 01 22:37:03.437931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.437898 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68"] Nov 01 22:37:03.665183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.646905 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7deebd9c-6143-45e1-934b-12bdf6cd416f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68\" (UID: \"7deebd9c-6143-45e1-934b-12bdf6cd416f\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:03.665183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.647025 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e867a1df-a0ee-496b-a331-3d6ade14bee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8\" (UID: \"e867a1df-a0ee-496b-a331-3d6ade14bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:03.692146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.647402 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgrqf\" (UniqueName: \"kubernetes.io/projected/7deebd9c-6143-45e1-934b-12bdf6cd416f-kube-api-access-sgrqf\") pod \"payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68\" (UID: \"7deebd9c-6143-45e1-934b-12bdf6cd416f\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:03.692146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.687340 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8rfb\" (UniqueName: \"kubernetes.io/projected/e867a1df-a0ee-496b-a331-3d6ade14bee2-kube-api-access-q8rfb\") pod \"payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8\" (UID: \"e867a1df-a0ee-496b-a331-3d6ade14bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:03.692146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.649900 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e867a1df-a0ee-496b-a331-3d6ade14bee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8\" (UID: \"e867a1df-a0ee-496b-a331-3d6ade14bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:03.735136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.729071 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8rfb\" (UniqueName: \"kubernetes.io/projected/e867a1df-a0ee-496b-a331-3d6ade14bee2-kube-api-access-q8rfb\") pod \"payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8\" (UID: \"e867a1df-a0ee-496b-a331-3d6ade14bee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:03.807178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.789238 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7deebd9c-6143-45e1-934b-12bdf6cd416f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68\" (UID: \"7deebd9c-6143-45e1-934b-12bdf6cd416f\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:03.807178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.789432 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgrqf\" (UniqueName: \"kubernetes.io/projected/7deebd9c-6143-45e1-934b-12bdf6cd416f-kube-api-access-sgrqf\") pod \"payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68\" (UID: \"7deebd9c-6143-45e1-934b-12bdf6cd416f\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:03.807939 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.807898 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7deebd9c-6143-45e1-934b-12bdf6cd416f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68\" (UID: \"7deebd9c-6143-45e1-934b-12bdf6cd416f\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:03.836620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:03.831516 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgrqf\" (UniqueName: \"kubernetes.io/projected/7deebd9c-6143-45e1-934b-12bdf6cd416f-kube-api-access-sgrqf\") pod \"payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68\" (UID: \"7deebd9c-6143-45e1-934b-12bdf6cd416f\") " pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:04.000617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.000527 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:04.042956 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.042907 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:04.215738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.215054 6466 generic.go:356] "Generic (PLEG): container finished" podID="28632ab8-53e2-4244-82e1-e0daa463be0d" containerID="70112b71f1a9c8a747cbfd6ffbe4e2ede4c9cb3fa726b16693178841760fea6e" exitCode=0 Nov 01 22:37:04.215738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.215504 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" event={"ID":"28632ab8-53e2-4244-82e1-e0daa463be0d","Type":"ContainerDied","Data":"70112b71f1a9c8a747cbfd6ffbe4e2ede4c9cb3fa726b16693178841760fea6e"} Nov 01 22:37:04.226688 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.226541 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" podUID="d2e22b11-3488-4ba5-b4cb-306be966e90d" containerName="payload-pre-processing" containerID="cri-o://000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d" gracePeriod=25 Nov 01 22:37:04.306159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.305044 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8"] Nov 01 22:37:04.373892 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:37:04.373856 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7deebd9c_6143_45e1_934b_12bdf6cd416f.slice/crio-45d4d7825095982800a458bc63b73364793c7665512528a0a6ff180a47880cc6 WatchSource:0}: Error finding container 45d4d7825095982800a458bc63b73364793c7665512528a0a6ff180a47880cc6: Status 404 returned error can't find the container with id 45d4d7825095982800a458bc63b73364793c7665512528a0a6ff180a47880cc6 Nov 01 22:37:04.377025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.376991 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68"] Nov 01 22:37:04.387375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.387352 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:37:04.511092 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511065 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-kube-api-access-4lkqb\" (UniqueName: \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-kube-api-access-4lkqb\") pod \"28632ab8-53e2-4244-82e1-e0daa463be0d\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " Nov 01 22:37:04.511330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511236 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-token\" (UniqueName: \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-token\") pod \"28632ab8-53e2-4244-82e1-e0daa463be0d\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " Nov 01 22:37:04.511330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511287 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-credential-socket\") pod \"28632ab8-53e2-4244-82e1-e0daa463be0d\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " Nov 01 22:37:04.511450 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511354 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-socket\") pod \"28632ab8-53e2-4244-82e1-e0daa463be0d\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " Nov 01 22:37:04.511450 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511384 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-envoy\") pod \"28632ab8-53e2-4244-82e1-e0daa463be0d\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " Nov 01 22:37:04.511450 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511430 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28632ab8-53e2-4244-82e1-e0daa463be0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28632ab8-53e2-4244-82e1-e0daa463be0d-istiod-ca-cert\") pod \"28632ab8-53e2-4244-82e1-e0daa463be0d\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " Nov 01 22:37:04.511606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511497 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-data\") pod \"28632ab8-53e2-4244-82e1-e0daa463be0d\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " Nov 01 22:37:04.511606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511565 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-podinfo\") pod \"28632ab8-53e2-4244-82e1-e0daa463be0d\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " Nov 01 22:37:04.511702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511668 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-certs\") pod \"28632ab8-53e2-4244-82e1-e0daa463be0d\" (UID: \"28632ab8-53e2-4244-82e1-e0daa463be0d\") " Nov 01 22:37:04.511860 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511801 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-credential-socket" pod "28632ab8-53e2-4244-82e1-e0daa463be0d" (UID: "28632ab8-53e2-4244-82e1-e0daa463be0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:04.511961 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511932 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-socket" pod "28632ab8-53e2-4244-82e1-e0daa463be0d" (UID: "28632ab8-53e2-4244-82e1-e0daa463be0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:04.512019 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.511993 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-data" pod "28632ab8-53e2-4244-82e1-e0daa463be0d" (UID: "28632ab8-53e2-4244-82e1-e0daa463be0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:04.512399 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.512374 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28632ab8-53e2-4244-82e1-e0daa463be0d-istiod-ca-cert" pod "28632ab8-53e2-4244-82e1-e0daa463be0d" (UID: "28632ab8-53e2-4244-82e1-e0daa463be0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:04.512518 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.512498 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-certs" pod "28632ab8-53e2-4244-82e1-e0daa463be0d" (UID: "28632ab8-53e2-4244-82e1-e0daa463be0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:04.512620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.512596 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:04.512677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.512626 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:04.512677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.512643 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28632ab8-53e2-4244-82e1-e0daa463be0d-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:04.512677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.512657 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:04.514740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.514706 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-podinfo" pod "28632ab8-53e2-4244-82e1-e0daa463be0d" (UID: "28632ab8-53e2-4244-82e1-e0daa463be0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:37:04.515005 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.514977 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-envoy" pod "28632ab8-53e2-4244-82e1-e0daa463be0d" (UID: "28632ab8-53e2-4244-82e1-e0daa463be0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:04.515083 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.515015 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-token" pod "28632ab8-53e2-4244-82e1-e0daa463be0d" (UID: "28632ab8-53e2-4244-82e1-e0daa463be0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:04.516864 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.516735 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-kube-api-access-4lkqb" pod "28632ab8-53e2-4244-82e1-e0daa463be0d" (UID: "28632ab8-53e2-4244-82e1-e0daa463be0d"). InnerVolumeSpecName "kube-api-access-4lkqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:04.567831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.567799 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:37:04.614983 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.614947 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:04.614983 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.614986 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:04.615202 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.615002 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28632ab8-53e2-4244-82e1-e0daa463be0d-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:04.615202 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.615016 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28632ab8-53e2-4244-82e1-e0daa463be0d-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:04.615202 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.615031 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lkqb\" (UniqueName: \"kubernetes.io/projected/28632ab8-53e2-4244-82e1-e0daa463be0d-kube-api-access-4lkqb\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:04.718435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.718395 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e22b11-3488-4ba5-b4cb-306be966e90d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e22b11-3488-4ba5-b4cb-306be966e90d-plugins-config-volume\") pod \"d2e22b11-3488-4ba5-b4cb-306be966e90d\" (UID: \"d2e22b11-3488-4ba5-b4cb-306be966e90d\") " Nov 01 22:37:04.718749 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.718722 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e22b11-3488-4ba5-b4cb-306be966e90d-kube-api-access-hnlc4\" (UniqueName: \"kubernetes.io/projected/d2e22b11-3488-4ba5-b4cb-306be966e90d-kube-api-access-hnlc4\") pod \"d2e22b11-3488-4ba5-b4cb-306be966e90d\" (UID: \"d2e22b11-3488-4ba5-b4cb-306be966e90d\") " Nov 01 22:37:04.721749 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.721363 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e22b11-3488-4ba5-b4cb-306be966e90d-plugins-config-volume" pod "d2e22b11-3488-4ba5-b4cb-306be966e90d" (UID: "d2e22b11-3488-4ba5-b4cb-306be966e90d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:04.723883 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.723852 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e22b11-3488-4ba5-b4cb-306be966e90d-kube-api-access-hnlc4" pod "d2e22b11-3488-4ba5-b4cb-306be966e90d" (UID: "d2e22b11-3488-4ba5-b4cb-306be966e90d"). InnerVolumeSpecName "kube-api-access-hnlc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:04.826235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.821605 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnlc4\" (UniqueName: \"kubernetes.io/projected/d2e22b11-3488-4ba5-b4cb-306be966e90d-kube-api-access-hnlc4\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:04.826235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:04.826200 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e22b11-3488-4ba5-b4cb-306be966e90d-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:05.225844 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.225797 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" event={"ID":"e867a1df-a0ee-496b-a331-3d6ade14bee2","Type":"ContainerStarted","Data":"f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1"} Nov 01 22:37:05.226043 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.225912 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:05.226043 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.225936 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" event={"ID":"e867a1df-a0ee-496b-a331-3d6ade14bee2","Type":"ContainerStarted","Data":"ef1b7617da159c5b3278ad344727e8ab64f9a513b067921dc9349ec8135142a5"} Nov 01 22:37:05.227448 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.227425 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" Nov 01 22:37:05.227573 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.227424 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8" event={"ID":"28632ab8-53e2-4244-82e1-e0daa463be0d","Type":"ContainerDied","Data":"64ead3e24798fbe641ee1de186292ae7ed03065cf6ae05982b68d82d7fde5cf2"} Nov 01 22:37:05.227625 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.227573 6466 scope.go:122] "RemoveContainer" containerID="70112b71f1a9c8a747cbfd6ffbe4e2ede4c9cb3fa726b16693178841760fea6e" Nov 01 22:37:05.229738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.229710 6466 generic.go:356] "Generic (PLEG): container finished" podID="d2e22b11-3488-4ba5-b4cb-306be966e90d" containerID="000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d" exitCode=0 Nov 01 22:37:05.229875 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.229795 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" event={"ID":"d2e22b11-3488-4ba5-b4cb-306be966e90d","Type":"ContainerDied","Data":"000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d"} Nov 01 22:37:05.229875 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.229804 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" Nov 01 22:37:05.229875 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.229838 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q" event={"ID":"d2e22b11-3488-4ba5-b4cb-306be966e90d","Type":"ContainerDied","Data":"a1a8da461d1a7118590d07f8b464a7064b15b9599a8ced48864c9ebd19740c61"} Nov 01 22:37:05.233041 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.233014 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" event={"ID":"7deebd9c-6143-45e1-934b-12bdf6cd416f","Type":"ContainerStarted","Data":"d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3"} Nov 01 22:37:05.233209 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.233065 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" event={"ID":"7deebd9c-6143-45e1-934b-12bdf6cd416f","Type":"ContainerStarted","Data":"45d4d7825095982800a458bc63b73364793c7665512528a0a6ff180a47880cc6"} Nov 01 22:37:05.233314 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.233248 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:05.240820 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.240797 6466 scope.go:122] "RemoveContainer" containerID="000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d" Nov 01 22:37:05.245913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.245870 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" podStartSLOduration=2.24585541 podStartE2EDuration="2.24585541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:05.243591651 +0000 UTC m=+829.028030572" watchObservedRunningTime="2026-11-01 22:37:05.24585541 +0000 UTC m=+829.030294331" Nov 01 22:37:05.253780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.253760 6466 scope.go:122] "RemoveContainer" containerID="000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d" Nov 01 22:37:05.254088 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:05.254059 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d\": container with ID starting with 000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d not found: ID does not exist" containerID="000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d" Nov 01 22:37:05.254192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.254088 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d"} err="failed to get container status \"000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d\": rpc error: code = NotFound desc = could not find container \"000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d\": container with ID starting with 000079c49686f0d4c19b86638ae8c169921fe07416181f19f0de07a5809ffa7d not found: ID does not exist" Nov 01 22:37:05.262205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.262177 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q"] Nov 01 22:37:05.265878 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.265861 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fd214936-5854759f59-46g2q"] Nov 01 22:37:05.280343 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.280301 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" podStartSLOduration=2.280289086 podStartE2EDuration="2.280289086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:05.277377233 +0000 UTC m=+829.061816167" watchObservedRunningTime="2026-11-01 22:37:05.280289086 +0000 UTC m=+829.064728016" Nov 01 22:37:05.297911 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.297885 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8"] Nov 01 22:37:05.306452 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.306428 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0500c01f-openshift-default-84f7c987bc-k52h8"] Nov 01 22:37:05.858795 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.858765 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9"] Nov 01 22:37:05.859277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:05.859009 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" podUID="3990f82b-1cdb-47d7-986f-2a20f096f3b6" containerName="maas-api" containerID="cri-o://7464c39bb32678e1b8ecdf076018432f5c88dd528a36906e65dc1f7af321148b" gracePeriod=30 Nov 01 22:37:06.675680 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.641492 6466 generic.go:356] "Generic (PLEG): container finished" podID="3990f82b-1cdb-47d7-986f-2a20f096f3b6" containerID="7464c39bb32678e1b8ecdf076018432f5c88dd528a36906e65dc1f7af321148b" exitCode=0 Nov 01 22:37:06.745756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.734088 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28632ab8-53e2-4244-82e1-e0daa463be0d" path="/var/lib/kubelet/pods/28632ab8-53e2-4244-82e1-e0daa463be0d/volumes" Nov 01 22:37:06.745756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.737454 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e22b11-3488-4ba5-b4cb-306be966e90d" path="/var/lib/kubelet/pods/d2e22b11-3488-4ba5-b4cb-306be966e90d/volumes" Nov 01 22:37:06.754040 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.750725 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" event={"ID":"3990f82b-1cdb-47d7-986f-2a20f096f3b6","Type":"ContainerDied","Data":"7464c39bb32678e1b8ecdf076018432f5c88dd528a36906e65dc1f7af321148b"} Nov 01 22:37:06.755793 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.755767 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:37:06.883261 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.883164 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-metrics-certs\") pod \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " Nov 01 22:37:06.883664 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.883376 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-maas-api-tls\") pod \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " Nov 01 22:37:06.883664 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.883477 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3990f82b-1cdb-47d7-986f-2a20f096f3b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3990f82b-1cdb-47d7-986f-2a20f096f3b6-openshift-service-ca\") pod \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " Nov 01 22:37:06.883664 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.883549 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3990f82b-1cdb-47d7-986f-2a20f096f3b6-kube-api-access-x7sd9\" (UniqueName: \"kubernetes.io/projected/3990f82b-1cdb-47d7-986f-2a20f096f3b6-kube-api-access-x7sd9\") pod \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\" (UID: \"3990f82b-1cdb-47d7-986f-2a20f096f3b6\") " Nov 01 22:37:06.883910 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.883883 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3990f82b-1cdb-47d7-986f-2a20f096f3b6-openshift-service-ca" pod "3990f82b-1cdb-47d7-986f-2a20f096f3b6" (UID: "3990f82b-1cdb-47d7-986f-2a20f096f3b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:06.884687 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.884655 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3990f82b-1cdb-47d7-986f-2a20f096f3b6-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:06.885588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.885557 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-metrics-certs" pod "3990f82b-1cdb-47d7-986f-2a20f096f3b6" (UID: "3990f82b-1cdb-47d7-986f-2a20f096f3b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:06.885588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.885578 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-maas-api-tls" pod "3990f82b-1cdb-47d7-986f-2a20f096f3b6" (UID: "3990f82b-1cdb-47d7-986f-2a20f096f3b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:06.885905 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.885889 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3990f82b-1cdb-47d7-986f-2a20f096f3b6-kube-api-access-x7sd9" pod "3990f82b-1cdb-47d7-986f-2a20f096f3b6" (UID: "3990f82b-1cdb-47d7-986f-2a20f096f3b6"). InnerVolumeSpecName "kube-api-access-x7sd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:06.987032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.986873 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7sd9\" (UniqueName: \"kubernetes.io/projected/3990f82b-1cdb-47d7-986f-2a20f096f3b6-kube-api-access-x7sd9\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:06.987032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.986916 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:06.987032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:06.986930 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3990f82b-1cdb-47d7-986f-2a20f096f3b6-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:07.172865 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:07.172838 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:37:07.622629 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:07.622547 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr"] Nov 01 22:37:07.622839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:07.622788 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" podUID="9d5c5d0c-3959-4b87-879e-b581dc09a168" containerName="istio-proxy" containerID="cri-o://184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867" gracePeriod=30 Nov 01 22:37:07.827290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:07.821165 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" event={"ID":"3990f82b-1cdb-47d7-986f-2a20f096f3b6","Type":"ContainerDied","Data":"f0916332b49057d2e0ea2c8e8a82e5a81059428dca786fcc3d164b057c4b6819"} Nov 01 22:37:07.827290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:07.821356 6466 scope.go:122] "RemoveContainer" containerID="7464c39bb32678e1b8ecdf076018432f5c88dd528a36906e65dc1f7af321148b" Nov 01 22:37:07.827290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:07.821263 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9" Nov 01 22:37:07.958404 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:07.958368 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9"] Nov 01 22:37:07.969131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:07.966082 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fd214936-d88cc8bb8-dlqz9"] Nov 01 22:37:08.031169 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.030981 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:37:08.452141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.431676 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3990f82b-1cdb-47d7-986f-2a20f096f3b6" path="/var/lib/kubelet/pods/3990f82b-1cdb-47d7-986f-2a20f096f3b6/volumes" Nov 01 22:37:08.731883 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.731743 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv"] Nov 01 22:37:08.733074 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.733041 6466 cpu_manager.go:414] "containerMap: removing container" podUID="28632ab8-53e2-4244-82e1-e0daa463be0d" containerName="istio-proxy" Nov 01 22:37:08.733074 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.733071 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28632ab8-53e2-4244-82e1-e0daa463be0d" containerName="istio-proxy" Nov 01 22:37:08.733292 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.733121 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d2e22b11-3488-4ba5-b4cb-306be966e90d" containerName="payload-pre-processing" Nov 01 22:37:08.733292 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.733131 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e22b11-3488-4ba5-b4cb-306be966e90d" containerName="payload-pre-processing" Nov 01 22:37:08.733292 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.733188 6466 cpu_manager.go:414] "containerMap: removing container" podUID="3990f82b-1cdb-47d7-986f-2a20f096f3b6" containerName="maas-api" Nov 01 22:37:08.733292 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.733195 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3990f82b-1cdb-47d7-986f-2a20f096f3b6" containerName="maas-api" Nov 01 22:37:08.733495 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.733403 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="28632ab8-53e2-4244-82e1-e0daa463be0d" containerName="istio-proxy" Nov 01 22:37:08.733495 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.733421 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e22b11-3488-4ba5-b4cb-306be966e90d" containerName="payload-pre-processing" Nov 01 22:37:08.733495 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.733438 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="3990f82b-1cdb-47d7-986f-2a20f096f3b6" containerName="maas-api" Nov 01 22:37:08.741236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.739354 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.753913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.753886 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e4b83ffd-openshift-default-dockercfg-jlcxs\"" Nov 01 22:37:08.761061 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.760238 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv"] Nov 01 22:37:08.783783 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.783749 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-envoy\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.783975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.783792 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-data\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.783975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.783881 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rknv2\" (UniqueName: \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-kube-api-access-rknv2\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.784100 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.783984 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-socket\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.784100 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.784040 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3ce252-0ab4-499d-abac-fed6118d1273-istiod-ca-cert\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.784100 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.784070 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-token\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.784289 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.784146 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-podinfo\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.784289 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.784248 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-credential-socket\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.784396 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.784288 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-certs\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.891127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.889596 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-credential-socket\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.891127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.889669 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-certs\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.891127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.889865 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-envoy\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.891127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.889910 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-data\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.891127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.889957 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rknv2\" (UniqueName: \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-kube-api-access-rknv2\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.891127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.890062 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-socket\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.891127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.890174 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3ce252-0ab4-499d-abac-fed6118d1273-istiod-ca-cert\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.891127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.890221 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-token\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.891127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.890276 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-podinfo\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.916143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.911358 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-data\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.920131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.918008 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-certs\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.949129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.945485 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-credential-socket\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.949129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.946954 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-podinfo\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.949129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.948574 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-socket\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.949627 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.949554 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-token\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.952153 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.952088 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-envoy\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:08.965129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:08.963152 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3ce252-0ab4-499d-abac-fed6118d1273-istiod-ca-cert\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:09.023008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:09.022887 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rknv2\" (UniqueName: \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-kube-api-access-rknv2\") pod \"e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:09.079718 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:09.079419 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:09.860207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:09.860170 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv"] Nov 01 22:37:09.868952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:09.868914 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:09.869032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:09.868988 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:09.869032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:09.869020 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:09.959715 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:09.959680 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" event={"ID":"fb3ce252-0ab4-499d-abac-fed6118d1273","Type":"ContainerStarted","Data":"613431851b55a7d24b7dfb75fe3c143aa5489b1a62feea91e40fb1a5d8633d44"} Nov 01 22:37:09.959867 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:09.959743 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" event={"ID":"fb3ce252-0ab4-499d-abac-fed6118d1273","Type":"ContainerStarted","Data":"df7578264be325597b4842c5ee8e7803e604c74077a5a3658e9ea386353c0eb8"} Nov 01 22:37:09.984545 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:09.984467 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" podStartSLOduration=1.984446376 podStartE2EDuration="1.984446376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:09.982796606 +0000 UTC m=+833.767235528" watchObservedRunningTime="2026-11-01 22:37:09.984446376 +0000 UTC m=+833.768885387" Nov 01 22:37:10.081037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:10.081006 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:10.082052 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:10.082027 6466 patch_prober.go:29] interesting pod/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 22:37:10.082213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:10.082079 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" podUID="fb3ce252-0ab4-499d-abac-fed6118d1273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 22:37:11.085771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.085738 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:11.378475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.378401 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz"] Nov 01 22:37:11.382283 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.382238 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:11.387324 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.387299 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-61ce6e07-serving-cert\"" Nov 01 22:37:11.393000 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.392972 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz"] Nov 01 22:37:11.413417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.413362 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-maas-api-tls\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:11.413814 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.413787 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-metrics-certs\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:11.413932 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.413865 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9wmj\" (UniqueName: \"kubernetes.io/projected/cc5375cc-c59d-4614-8137-5782f114cb1f-kube-api-access-g9wmj\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:11.413932 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.413922 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc5375cc-c59d-4614-8137-5782f114cb1f-openshift-service-ca\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:11.515813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.515781 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-metrics-certs\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:11.515969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.515835 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wmj\" (UniqueName: \"kubernetes.io/projected/cc5375cc-c59d-4614-8137-5782f114cb1f-kube-api-access-g9wmj\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:11.515969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.515888 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc5375cc-c59d-4614-8137-5782f114cb1f-openshift-service-ca\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:11.516052 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.516039 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-maas-api-tls\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:11.516218 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:11.516199 6466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-serving-cert: secret "maas-api-e2e-migrate-61ce6e07-serving-cert" not found Nov 01 22:37:11.516287 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:11.516276 6466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-maas-api-tls podName:cc5375cc-c59d-4614-8137-5782f114cb1f nodeName:}" failed. No retries permitted until 2026-11-01 22:37:12.016257109 +0000 UTC m=+835.800696049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-maas-api-tls") pod "maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" (UID: "cc5375cc-c59d-4614-8137-5782f114cb1f") : secret "maas-api-e2e-migrate-61ce6e07-serving-cert" not found Nov 01 22:37:11.516621 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.516599 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc5375cc-c59d-4614-8137-5782f114cb1f-openshift-service-ca\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:11.518446 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.518421 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-metrics-certs\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:11.531734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:11.531707 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wmj\" (UniqueName: \"kubernetes.io/projected/cc5375cc-c59d-4614-8137-5782f114cb1f-kube-api-access-g9wmj\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:12.060135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:12.059456 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-maas-api-tls\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:12.068897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:12.066086 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:12.068897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:12.067154 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:12.071239 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:12.071211 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-maas-api-tls\") pod \"maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:12.300588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:12.300528 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:12.818846 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:12.818663 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz"] Nov 01 22:37:12.916802 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:12.916775 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:37:13.032181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.032150 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6"] Nov 01 22:37:13.032985 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.032969 6466 cpu_manager.go:414] "containerMap: removing container" podUID="9d5c5d0c-3959-4b87-879e-b581dc09a168" containerName="istio-proxy" Nov 01 22:37:13.033033 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.032986 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5c5d0c-3959-4b87-879e-b581dc09a168" containerName="istio-proxy" Nov 01 22:37:13.033144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.033123 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5c5d0c-3959-4b87-879e-b581dc09a168" containerName="istio-proxy" Nov 01 22:37:13.035225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035201 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-credential-socket\") pod \"9d5c5d0c-3959-4b87-879e-b581dc09a168\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " Nov 01 22:37:13.035317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035297 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-data\") pod \"9d5c5d0c-3959-4b87-879e-b581dc09a168\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " Nov 01 22:37:13.035375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035331 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-certs\") pod \"9d5c5d0c-3959-4b87-879e-b581dc09a168\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " Nov 01 22:37:13.035417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035376 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5c5d0c-3959-4b87-879e-b581dc09a168-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5c5d0c-3959-4b87-879e-b581dc09a168-istiod-ca-cert\") pod \"9d5c5d0c-3959-4b87-879e-b581dc09a168\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " Nov 01 22:37:13.035503 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035465 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-credential-socket" pod "9d5c5d0c-3959-4b87-879e-b581dc09a168" (UID: "9d5c5d0c-3959-4b87-879e-b581dc09a168"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:13.035503 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035491 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-token\" (UniqueName: \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-token\") pod \"9d5c5d0c-3959-4b87-879e-b581dc09a168\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " Nov 01 22:37:13.035659 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035567 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-kube-api-access-27zlq\" (UniqueName: \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-kube-api-access-27zlq\") pod \"9d5c5d0c-3959-4b87-879e-b581dc09a168\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " Nov 01 22:37:13.035659 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035585 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-data" pod "9d5c5d0c-3959-4b87-879e-b581dc09a168" (UID: "9d5c5d0c-3959-4b87-879e-b581dc09a168"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:13.035659 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035620 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-socket\") pod \"9d5c5d0c-3959-4b87-879e-b581dc09a168\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " Nov 01 22:37:13.035813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035663 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-certs" pod "9d5c5d0c-3959-4b87-879e-b581dc09a168" (UID: "9d5c5d0c-3959-4b87-879e-b581dc09a168"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:13.035813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035757 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-envoy\") pod \"9d5c5d0c-3959-4b87-879e-b581dc09a168\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " Nov 01 22:37:13.035813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035803 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5c5d0c-3959-4b87-879e-b581dc09a168-istiod-ca-cert" pod "9d5c5d0c-3959-4b87-879e-b581dc09a168" (UID: "9d5c5d0c-3959-4b87-879e-b581dc09a168"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:13.036010 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035900 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-socket" pod "9d5c5d0c-3959-4b87-879e-b581dc09a168" (UID: "9d5c5d0c-3959-4b87-879e-b581dc09a168"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:13.036010 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035906 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-podinfo\") pod \"9d5c5d0c-3959-4b87-879e-b581dc09a168\" (UID: \"9d5c5d0c-3959-4b87-879e-b581dc09a168\") " Nov 01 22:37:13.036010 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.035926 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:13.036944 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.036918 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.037025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.036950 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.037025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.036965 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.037025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.036979 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d5c5d0c-3959-4b87-879e-b581dc09a168-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.037025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.036993 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.038065 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.038035 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-envoy" pod "9d5c5d0c-3959-4b87-879e-b581dc09a168" (UID: "9d5c5d0c-3959-4b87-879e-b581dc09a168"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:13.038203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.038170 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-podinfo" pod "9d5c5d0c-3959-4b87-879e-b581dc09a168" (UID: "9d5c5d0c-3959-4b87-879e-b581dc09a168"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:37:13.038251 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.038203 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-token" pod "9d5c5d0c-3959-4b87-879e-b581dc09a168" (UID: "9d5c5d0c-3959-4b87-879e-b581dc09a168"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:13.039003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.038976 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-kube-api-access-27zlq" pod "9d5c5d0c-3959-4b87-879e-b581dc09a168" (UID: "9d5c5d0c-3959-4b87-879e-b581dc09a168"). InnerVolumeSpecName "kube-api-access-27zlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:13.049054 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.048891 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e4b83ffd\"" Nov 01 22:37:13.055388 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.055358 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6"] Nov 01 22:37:13.085084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.085055 6466 generic.go:356] "Generic (PLEG): container finished" podID="9d5c5d0c-3959-4b87-879e-b581dc09a168" containerID="184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867" exitCode=0 Nov 01 22:37:13.085260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.085146 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" Nov 01 22:37:13.085260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.085150 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" event={"ID":"9d5c5d0c-3959-4b87-879e-b581dc09a168","Type":"ContainerDied","Data":"184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867"} Nov 01 22:37:13.085260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.085221 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr" event={"ID":"9d5c5d0c-3959-4b87-879e-b581dc09a168","Type":"ContainerDied","Data":"a939c1ca9e23fedf0ff5dd7170cd1957512dab1073b66a69c990e3b291cc4688"} Nov 01 22:37:13.085260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.085251 6466 scope.go:122] "RemoveContainer" containerID="184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867" Nov 01 22:37:13.087369 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.087193 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" event={"ID":"cc5375cc-c59d-4614-8137-5782f114cb1f","Type":"ContainerStarted","Data":"6a76c82f4e129f6976531c94a26ed5c1c675a03b86323ba5e07d61fc45f7efe4"} Nov 01 22:37:13.099897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.099875 6466 scope.go:122] "RemoveContainer" containerID="184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867" Nov 01 22:37:13.100220 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:13.100200 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867\": container with ID starting with 184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867 not found: ID does not exist" containerID="184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867" Nov 01 22:37:13.100282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.100227 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867"} err="failed to get container status \"184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867\": rpc error: code = NotFound desc = could not find container \"184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867\": container with ID starting with 184d38f5b0d346c30ae42c6c56fa5fd0969e515d711fb90380c2702e2095b867 not found: ID does not exist" Nov 01 22:37:13.108981 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.108959 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm"] Nov 01 22:37:13.112821 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.112806 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:13.121014 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.120990 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e4b83ffd-dockercfg-7b46m\"" Nov 01 22:37:13.121235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.121212 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr"] Nov 01 22:37:13.126453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.126416 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm"] Nov 01 22:37:13.130549 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.130525 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fd214936-openshift-default-847dfbd9c9-pnvvr"] Nov 01 22:37:13.138146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.138120 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhcq\" (UniqueName: \"kubernetes.io/projected/19aa924b-c7f5-4622-9c6f-865fc6e33897-kube-api-access-6nhcq\") pod \"payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6\" (UID: \"19aa924b-c7f5-4622-9c6f-865fc6e33897\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:13.138418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.138395 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19aa924b-c7f5-4622-9c6f-865fc6e33897-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6\" (UID: \"19aa924b-c7f5-4622-9c6f-865fc6e33897\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:13.138546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.138531 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27zlq\" (UniqueName: \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-kube-api-access-27zlq\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.138606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.138552 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.138606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.138566 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.138606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.138580 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d5c5d0c-3959-4b87-879e-b581dc09a168-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.359380 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.359290 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhcq\" (UniqueName: \"kubernetes.io/projected/19aa924b-c7f5-4622-9c6f-865fc6e33897-kube-api-access-6nhcq\") pod \"payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6\" (UID: \"19aa924b-c7f5-4622-9c6f-865fc6e33897\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:13.478213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.478168 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9gs2\" (UniqueName: \"kubernetes.io/projected/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-kube-api-access-l9gs2\") pod \"payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm\" (UID: \"0321f7c9-74fa-470a-87e5-88e1f9b21ca1\") " pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:13.478878 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.478850 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm\" (UID: \"0321f7c9-74fa-470a-87e5-88e1f9b21ca1\") " pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:13.479621 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.479594 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19aa924b-c7f5-4622-9c6f-865fc6e33897-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6\" (UID: \"19aa924b-c7f5-4622-9c6f-865fc6e33897\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:13.481467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.480561 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19aa924b-c7f5-4622-9c6f-865fc6e33897-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6\" (UID: \"19aa924b-c7f5-4622-9c6f-865fc6e33897\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:13.574030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.564466 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhcq\" (UniqueName: \"kubernetes.io/projected/19aa924b-c7f5-4622-9c6f-865fc6e33897-kube-api-access-6nhcq\") pod \"payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6\" (UID: \"19aa924b-c7f5-4622-9c6f-865fc6e33897\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:13.586712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.584145 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9gs2\" (UniqueName: \"kubernetes.io/projected/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-kube-api-access-l9gs2\") pod \"payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm\" (UID: \"0321f7c9-74fa-470a-87e5-88e1f9b21ca1\") " pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:13.586712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.584222 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm\" (UID: \"0321f7c9-74fa-470a-87e5-88e1f9b21ca1\") " pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:13.618227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.618132 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm\" (UID: \"0321f7c9-74fa-470a-87e5-88e1f9b21ca1\") " pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:13.666281 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.666190 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9gs2\" (UniqueName: \"kubernetes.io/projected/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-kube-api-access-l9gs2\") pod \"payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm\" (UID: \"0321f7c9-74fa-470a-87e5-88e1f9b21ca1\") " pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:13.685012 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.684732 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:13.737085 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:13.737055 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:14.263887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:14.262796 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:14.292602 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:14.292552 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" podStartSLOduration=2.594277424 podStartE2EDuration="3.292537033s" totalImagesPullingTime="698.259609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:14.288017036 +0000 UTC m=+838.072455958" watchObservedRunningTime="2026-11-01 22:37:14.292537033 +0000 UTC m=+838.076975982" Nov 01 22:37:14.347784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:14.347745 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5c5d0c-3959-4b87-879e-b581dc09a168" path="/var/lib/kubelet/pods/9d5c5d0c-3959-4b87-879e-b581dc09a168/volumes" Nov 01 22:37:14.531853 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:37:14.525495 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0321f7c9_74fa_470a_87e5_88e1f9b21ca1.slice/crio-e4befdb3731e62ab061ac6a95d1c6afbfc4268f84422df22bdb6fd6858846eed WatchSource:0}: Error finding container e4befdb3731e62ab061ac6a95d1c6afbfc4268f84422df22bdb6fd6858846eed: Status 404 returned error can't find the container with id e4befdb3731e62ab061ac6a95d1c6afbfc4268f84422df22bdb6fd6858846eed Nov 01 22:37:14.531853 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:14.530550 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm"] Nov 01 22:37:14.538657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:14.538617 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6"] Nov 01 22:37:15.393141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:15.393081 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" event={"ID":"19aa924b-c7f5-4622-9c6f-865fc6e33897","Type":"ContainerStarted","Data":"d9a6e26d15508ac2a15f913b415b00104bfffca5b099fc8894d46b6c16869dd6"} Nov 01 22:37:15.393429 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:15.393403 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" event={"ID":"19aa924b-c7f5-4622-9c6f-865fc6e33897","Type":"ContainerStarted","Data":"5abdaef52c22e023c7ae51f89c497ba9503f2ad616ba11fddabe709a432dcb30"} Nov 01 22:37:15.420133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:15.394730 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:15.455805 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:15.455762 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" event={"ID":"cc5375cc-c59d-4614-8137-5782f114cb1f","Type":"ContainerStarted","Data":"00e7a0409e94663c8a732a08354cd18b8114d0e87e94436724123c781a7d7ca9"} Nov 01 22:37:15.520213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:15.494882 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" event={"ID":"0321f7c9-74fa-470a-87e5-88e1f9b21ca1","Type":"ContainerStarted","Data":"ba86b646d37a67ac6a49e8472e005bc71fe526464a8dfe3ce938c19a3105ed1d"} Nov 01 22:37:15.520213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:15.494949 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" event={"ID":"0321f7c9-74fa-470a-87e5-88e1f9b21ca1","Type":"ContainerStarted","Data":"e4befdb3731e62ab061ac6a95d1c6afbfc4268f84422df22bdb6fd6858846eed"} Nov 01 22:37:15.520213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:15.500530 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:15.615133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:15.589031 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" podStartSLOduration=2.589006983 podStartE2EDuration="2.589006983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:15.454361117 +0000 UTC m=+839.238800038" watchObservedRunningTime="2026-11-01 22:37:15.589006983 +0000 UTC m=+839.373445918" Nov 01 22:37:15.617703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:15.617646 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" podStartSLOduration=2.617632398 podStartE2EDuration="2.617632398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:15.549660114 +0000 UTC m=+839.334099048" watchObservedRunningTime="2026-11-01 22:37:15.617632398 +0000 UTC m=+839.402071322" Nov 01 22:37:15.953400 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:15.953323 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:16.190130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:16.189208 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:37:16.190130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:16.189688 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 01 22:37:16.305263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:16.300644 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:16.626191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:16.623803 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:16.698851 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:16.691050 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:16.778145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:16.770333 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:16.854362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:16.854327 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:16.917834 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:16.917786 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Nov 01 22:37:17.232197 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:17.223778 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:17.232197 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:17.226649 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:37:17.369137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:17.361975 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:17.369137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:17.362866 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:37:17.650953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:17.645312 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:37:17.650953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:17.646065 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:37:17.817243 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:17.817192 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:17.866134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:17.865437 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:37:18.288134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:18.287706 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:37:18.394797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:18.393813 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:18.654241 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:18.654199 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:18.839556 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:18.839354 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:37:18.839556 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:18.839419 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:18.848169 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:18.847752 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:19.062852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:19.062825 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:37:19.290316 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:19.290217 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:19.370133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:19.363480 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:37:19.421370 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:19.414218 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:37:19.421451 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:19.421423 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:37:19.433196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:19.432562 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:37:19.918338 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:19.918305 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47"] Nov 01 22:37:21.393125 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.393080 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz"] Nov 01 22:37:21.399477 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.399445 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:21.412445 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.412076 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz"] Nov 01 22:37:21.412445 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.412431 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e4b83ffd-serving-cert\"" Nov 01 22:37:21.538646 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.538607 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-metrics-certs\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:21.539702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.539668 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/629d4954-f71d-4354-b29c-d660b2fc922b-openshift-service-ca\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:21.540318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.540218 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gk9v\" (UniqueName: \"kubernetes.io/projected/629d4954-f71d-4354-b29c-d660b2fc922b-kube-api-access-7gk9v\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:21.540777 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.540755 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-maas-api-tls\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:21.626365 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.626327 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:21.673224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.671234 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-metrics-certs\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:21.673224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.671486 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/629d4954-f71d-4354-b29c-d660b2fc922b-openshift-service-ca\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:21.673224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.671725 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gk9v\" (UniqueName: \"kubernetes.io/projected/629d4954-f71d-4354-b29c-d660b2fc922b-kube-api-access-7gk9v\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:21.673224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.671828 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-maas-api-tls\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:21.673224 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:21.672056 6466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-serving-cert: secret "maas-api-e2e-ait-e4b83ffd-serving-cert" not found Nov 01 22:37:21.673224 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:21.672141 6466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-maas-api-tls podName:629d4954-f71d-4354-b29c-d660b2fc922b nodeName:}" failed. No retries permitted until 2026-11-01 22:37:22.17210081 +0000 UTC m=+845.956539725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-maas-api-tls") pod "maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" (UID: "629d4954-f71d-4354-b29c-d660b2fc922b") : secret "maas-api-e2e-ait-e4b83ffd-serving-cert" not found Nov 01 22:37:21.681550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.676027 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/629d4954-f71d-4354-b29c-d660b2fc922b-openshift-service-ca\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:21.683360 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.683333 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-metrics-certs\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:21.685062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:21.684783 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gk9v\" (UniqueName: \"kubernetes.io/projected/629d4954-f71d-4354-b29c-d660b2fc922b-kube-api-access-7gk9v\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:22.192255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:22.192214 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-maas-api-tls\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:22.250148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:22.249613 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-maas-api-tls\") pod \"maas-api-e2e-ait-e4b83ffd-5498577445-qqthz\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:22.319144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:22.318938 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:22.725867 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:22.725810 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz"] Nov 01 22:37:23.714431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:23.714399 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" event={"ID":"629d4954-f71d-4354-b29c-d660b2fc922b","Type":"ContainerStarted","Data":"c3fc83c1b8311084ca2b00f79095b78374d9e3b5f51a468827802700042c16b3"} Nov 01 22:37:23.714654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:23.714460 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" event={"ID":"629d4954-f71d-4354-b29c-d660b2fc922b","Type":"ContainerStarted","Data":"580ce90bbd612ddc00ddcf76828d20091f24aa30e5570fe78e351d17c86f3836"} Nov 01 22:37:23.714654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:23.714518 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:23.736512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:23.736460 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" podStartSLOduration=2.280626085 podStartE2EDuration="2.736444582s" totalImagesPullingTime="455.818497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:23.734721253 +0000 UTC m=+847.519160175" watchObservedRunningTime="2026-11-01 22:37:23.736444582 +0000 UTC m=+847.520883528" Nov 01 22:37:24.919616 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:24.919576 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" podUID="378b04d2-a448-4652-967c-c9c69071da63" containerName="payload-pre-processing" containerID="cri-o://0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687" gracePeriod=25 Nov 01 22:37:25.371522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.371483 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:37:25.442954 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.442920 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378b04d2-a448-4652-967c-c9c69071da63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378b04d2-a448-4652-967c-c9c69071da63-plugins-config-volume\") pod \"378b04d2-a448-4652-967c-c9c69071da63\" (UID: \"378b04d2-a448-4652-967c-c9c69071da63\") " Nov 01 22:37:25.443171 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.443129 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378b04d2-a448-4652-967c-c9c69071da63-kube-api-access-5fbb9\" (UniqueName: \"kubernetes.io/projected/378b04d2-a448-4652-967c-c9c69071da63-kube-api-access-5fbb9\") pod \"378b04d2-a448-4652-967c-c9c69071da63\" (UID: \"378b04d2-a448-4652-967c-c9c69071da63\") " Nov 01 22:37:25.443402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.443376 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378b04d2-a448-4652-967c-c9c69071da63-plugins-config-volume" pod "378b04d2-a448-4652-967c-c9c69071da63" (UID: "378b04d2-a448-4652-967c-c9c69071da63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:25.444074 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.444045 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/378b04d2-a448-4652-967c-c9c69071da63-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:25.445589 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.445559 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378b04d2-a448-4652-967c-c9c69071da63-kube-api-access-5fbb9" pod "378b04d2-a448-4652-967c-c9c69071da63" (UID: "378b04d2-a448-4652-967c-c9c69071da63"). InnerVolumeSpecName "kube-api-access-5fbb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:25.545521 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.545490 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fbb9\" (UniqueName: \"kubernetes.io/projected/378b04d2-a448-4652-967c-c9c69071da63-kube-api-access-5fbb9\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:25.729730 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.729697 6466 generic.go:356] "Generic (PLEG): container finished" podID="378b04d2-a448-4652-967c-c9c69071da63" containerID="0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687" exitCode=0 Nov 01 22:37:25.729906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.729744 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" Nov 01 22:37:25.729906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.729776 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" event={"ID":"378b04d2-a448-4652-967c-c9c69071da63","Type":"ContainerDied","Data":"0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687"} Nov 01 22:37:25.729906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.729831 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47" event={"ID":"378b04d2-a448-4652-967c-c9c69071da63","Type":"ContainerDied","Data":"1afba2397004d8554a3dfe7556a3ffba655abca43520fa10cae0dcc485f68e0c"} Nov 01 22:37:25.729906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.729853 6466 scope.go:122] "RemoveContainer" containerID="0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687" Nov 01 22:37:25.740658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.740640 6466 scope.go:122] "RemoveContainer" containerID="0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687" Nov 01 22:37:25.740935 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:25.740916 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687\": container with ID starting with 0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687 not found: ID does not exist" containerID="0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687" Nov 01 22:37:25.741008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.740942 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687"} err="failed to get container status \"0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687\": rpc error: code = NotFound desc = could not find container \"0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687\": container with ID starting with 0c3764bb5129278974ce5621a8115bb26f1fcb8cae3ea9a258656734aa34f687 not found: ID does not exist" Nov 01 22:37:25.753810 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.753762 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47"] Nov 01 22:37:25.757773 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:25.757749 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26aaf165-6bd66d979f-26m47"] Nov 01 22:37:26.347190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:26.347158 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378b04d2-a448-4652-967c-c9c69071da63" path="/var/lib/kubelet/pods/378b04d2-a448-4652-967c-c9c69071da63/volumes" Nov 01 22:37:28.484913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:28.484879 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:28.509299 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:28.509266 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:28.592168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:28.592135 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4"] Nov 01 22:37:28.592948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:28.592576 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" podUID="0e7333c3-94fd-4589-875b-43d81c90a0be" containerName="maas-api" containerID="cri-o://70492cca2f4ae70939b23876e9e80038fe3b53d509d19dfbb864f2a9c14b4d9b" gracePeriod=30 Nov 01 22:37:29.776882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:29.731795 6466 generic.go:356] "Generic (PLEG): container finished" podID="0e7333c3-94fd-4589-875b-43d81c90a0be" containerID="70492cca2f4ae70939b23876e9e80038fe3b53d509d19dfbb864f2a9c14b4d9b" exitCode=0 Nov 01 22:37:29.855580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:29.841701 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" event={"ID":"0e7333c3-94fd-4589-875b-43d81c90a0be","Type":"ContainerDied","Data":"70492cca2f4ae70939b23876e9e80038fe3b53d509d19dfbb864f2a9c14b4d9b"} Nov 01 22:37:29.855580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:29.841835 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:29.862303 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:29.862275 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:37:30.010950 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.010920 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e7333c3-94fd-4589-875b-43d81c90a0be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7333c3-94fd-4589-875b-43d81c90a0be-openshift-service-ca\") pod \"0e7333c3-94fd-4589-875b-43d81c90a0be\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " Nov 01 22:37:30.011151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.010961 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7333c3-94fd-4589-875b-43d81c90a0be-kube-api-access-ks9br\" (UniqueName: \"kubernetes.io/projected/0e7333c3-94fd-4589-875b-43d81c90a0be-kube-api-access-ks9br\") pod \"0e7333c3-94fd-4589-875b-43d81c90a0be\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " Nov 01 22:37:30.011151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.010998 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-maas-api-tls\") pod \"0e7333c3-94fd-4589-875b-43d81c90a0be\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " Nov 01 22:37:30.011235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.011215 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-metrics-certs\") pod \"0e7333c3-94fd-4589-875b-43d81c90a0be\" (UID: \"0e7333c3-94fd-4589-875b-43d81c90a0be\") " Nov 01 22:37:30.011420 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.011390 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7333c3-94fd-4589-875b-43d81c90a0be-openshift-service-ca" pod "0e7333c3-94fd-4589-875b-43d81c90a0be" (UID: "0e7333c3-94fd-4589-875b-43d81c90a0be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:30.012397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.012358 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7333c3-94fd-4589-875b-43d81c90a0be-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:30.013537 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.013511 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-maas-api-tls" pod "0e7333c3-94fd-4589-875b-43d81c90a0be" (UID: "0e7333c3-94fd-4589-875b-43d81c90a0be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:30.013537 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.013530 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-metrics-certs" pod "0e7333c3-94fd-4589-875b-43d81c90a0be" (UID: "0e7333c3-94fd-4589-875b-43d81c90a0be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:30.013680 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.013538 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7333c3-94fd-4589-875b-43d81c90a0be-kube-api-access-ks9br" pod "0e7333c3-94fd-4589-875b-43d81c90a0be" (UID: "0e7333c3-94fd-4589-875b-43d81c90a0be"). InnerVolumeSpecName "kube-api-access-ks9br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:30.113873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.113839 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks9br\" (UniqueName: \"kubernetes.io/projected/0e7333c3-94fd-4589-875b-43d81c90a0be-kube-api-access-ks9br\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:30.113873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.113869 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:30.113873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.113879 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e7333c3-94fd-4589-875b-43d81c90a0be-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:30.741169 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.741075 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" Nov 01 22:37:30.741169 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.741092 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4" event={"ID":"0e7333c3-94fd-4589-875b-43d81c90a0be","Type":"ContainerDied","Data":"bf88f4e7381151a6e3fba3c2714b75f3f7adf54fedf66a080d97279fd0364ca7"} Nov 01 22:37:30.741371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.741182 6466 scope.go:122] "RemoveContainer" containerID="70492cca2f4ae70939b23876e9e80038fe3b53d509d19dfbb864f2a9c14b4d9b" Nov 01 22:37:30.773369 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.773341 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4"] Nov 01 22:37:30.781469 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.781441 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26aaf165-5879bf6bdb-j26c4"] Nov 01 22:37:30.788425 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.788399 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98"] Nov 01 22:37:30.789170 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.789154 6466 cpu_manager.go:414] "containerMap: removing container" podUID="0e7333c3-94fd-4589-875b-43d81c90a0be" containerName="maas-api" Nov 01 22:37:30.789230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.789173 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7333c3-94fd-4589-875b-43d81c90a0be" containerName="maas-api" Nov 01 22:37:30.789230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.789212 6466 cpu_manager.go:414] "containerMap: removing container" podUID="378b04d2-a448-4652-967c-c9c69071da63" containerName="payload-pre-processing" Nov 01 22:37:30.789230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.789220 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378b04d2-a448-4652-967c-c9c69071da63" containerName="payload-pre-processing" Nov 01 22:37:30.789375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.789366 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="378b04d2-a448-4652-967c-c9c69071da63" containerName="payload-pre-processing" Nov 01 22:37:30.789411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.789386 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7333c3-94fd-4589-875b-43d81c90a0be" containerName="maas-api" Nov 01 22:37:30.794864 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.794845 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.799774 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.799733 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-66033bd2-openshift-default-dockercfg-mz5qb\"" Nov 01 22:37:30.805627 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.805525 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98"] Nov 01 22:37:30.867663 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.867630 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-credential-socket\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.867663 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.867667 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-socket\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.867888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.867839 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-istio-token\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.867962 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.867947 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxdfk\" (UniqueName: \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-kube-api-access-mxdfk\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.868009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.867977 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-certs\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.868055 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.868035 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80391b86-89a6-41db-8baa-59c62315596b-istio-podinfo\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.868214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.868185 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-data\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.868297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.868246 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80391b86-89a6-41db-8baa-59c62315596b-istiod-ca-cert\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.868297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.868276 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-envoy\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.969388 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.969349 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxdfk\" (UniqueName: \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-kube-api-access-mxdfk\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.969388 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.969390 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-certs\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.969711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.969410 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80391b86-89a6-41db-8baa-59c62315596b-istio-podinfo\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.969711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.969536 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-data\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.969711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.969607 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80391b86-89a6-41db-8baa-59c62315596b-istiod-ca-cert\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.969711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.969646 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-envoy\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.969711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.969704 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-credential-socket\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.969978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.969736 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-socket\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.969978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.969834 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-certs\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.969978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.969861 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-istio-token\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.969978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.969905 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-data\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.970198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.970049 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-credential-socket\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.970198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.970124 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-socket\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.970466 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.970444 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80391b86-89a6-41db-8baa-59c62315596b-istiod-ca-cert\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.972008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.971979 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80391b86-89a6-41db-8baa-59c62315596b-istio-podinfo\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.972087 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.972076 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-envoy\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.982210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.982179 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-istio-token\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:30.982326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:30.982310 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxdfk\" (UniqueName: \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-kube-api-access-mxdfk\") pod \"e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:31.115934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:31.115489 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:31.849206 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:31.849170 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98"] Nov 01 22:37:31.849927 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:37:31.849897 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80391b86_89a6_41db_8baa_59c62315596b.slice/crio-aef6208e7add18b2b498aff0cff6d97f33bdb6147d98c18c3d4ce1a8d68aedf4 WatchSource:0}: Error finding container aef6208e7add18b2b498aff0cff6d97f33bdb6147d98c18c3d4ce1a8d68aedf4: Status 404 returned error can't find the container with id aef6208e7add18b2b498aff0cff6d97f33bdb6147d98c18c3d4ce1a8d68aedf4 Nov 01 22:37:31.852847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:31.852815 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:31.852949 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:31.852885 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:31.852949 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:31.852913 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:32.343848 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:32.343811 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7333c3-94fd-4589-875b-43d81c90a0be" path="/var/lib/kubelet/pods/0e7333c3-94fd-4589-875b-43d81c90a0be/volumes" Nov 01 22:37:32.763290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:32.763260 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" event={"ID":"80391b86-89a6-41db-8baa-59c62315596b","Type":"ContainerStarted","Data":"4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8"} Nov 01 22:37:32.763467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:32.763310 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" event={"ID":"80391b86-89a6-41db-8baa-59c62315596b","Type":"ContainerStarted","Data":"aef6208e7add18b2b498aff0cff6d97f33bdb6147d98c18c3d4ce1a8d68aedf4"} Nov 01 22:37:32.791990 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:32.791935 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podStartSLOduration=2.791921126 podStartE2EDuration="2.791921126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:32.786254797 +0000 UTC m=+856.570693719" watchObservedRunningTime="2026-11-01 22:37:32.791921126 +0000 UTC m=+856.576360044" Nov 01 22:37:33.136183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:33.130153 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:33.147540 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:33.147409 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:33.819988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:33.819958 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:33.821958 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:33.821934 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:37:36.381901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.381865 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4"] Nov 01 22:37:36.386806 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.386789 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:37:36.393810 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.393787 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-66033bd2\"" Nov 01 22:37:36.397847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.397821 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4"] Nov 01 22:37:36.418102 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.418072 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7"] Nov 01 22:37:36.432932 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.432903 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:37:36.435219 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.435173 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7"] Nov 01 22:37:36.437290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.437264 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-66033bd2-dockercfg-cm5hz\"" Nov 01 22:37:36.558145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.557126 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8pzc\" (UniqueName: \"kubernetes.io/projected/3662ca88-be02-42ef-aadd-bce5e426d8bb-kube-api-access-c8pzc\") pod \"payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7\" (UID: \"3662ca88-be02-42ef-aadd-bce5e426d8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:37:36.558145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.557221 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d2qb\" (UniqueName: \"kubernetes.io/projected/54f64bb1-139c-453e-9c86-d7289bbcbf44-kube-api-access-4d2qb\") pod \"payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4\" (UID: \"54f64bb1-139c-453e-9c86-d7289bbcbf44\") " pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:37:36.558145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.557397 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3662ca88-be02-42ef-aadd-bce5e426d8bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7\" (UID: \"3662ca88-be02-42ef-aadd-bce5e426d8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:37:36.558145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.557481 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f64bb1-139c-453e-9c86-d7289bbcbf44-plugins-config-volume\") pod \"payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4\" (UID: \"54f64bb1-139c-453e-9c86-d7289bbcbf44\") " pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:37:36.664251 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.660286 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8pzc\" (UniqueName: \"kubernetes.io/projected/3662ca88-be02-42ef-aadd-bce5e426d8bb-kube-api-access-c8pzc\") pod \"payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7\" (UID: \"3662ca88-be02-42ef-aadd-bce5e426d8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:37:36.664570 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.664543 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d2qb\" (UniqueName: \"kubernetes.io/projected/54f64bb1-139c-453e-9c86-d7289bbcbf44-kube-api-access-4d2qb\") pod \"payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4\" (UID: \"54f64bb1-139c-453e-9c86-d7289bbcbf44\") " pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:37:36.664877 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.664856 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3662ca88-be02-42ef-aadd-bce5e426d8bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7\" (UID: \"3662ca88-be02-42ef-aadd-bce5e426d8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:37:36.691520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.691486 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f64bb1-139c-453e-9c86-d7289bbcbf44-plugins-config-volume\") pod \"payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4\" (UID: \"54f64bb1-139c-453e-9c86-d7289bbcbf44\") " pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:37:36.692873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.692516 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f64bb1-139c-453e-9c86-d7289bbcbf44-plugins-config-volume\") pod \"payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4\" (UID: \"54f64bb1-139c-453e-9c86-d7289bbcbf44\") " pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:37:36.693059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.666450 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3662ca88-be02-42ef-aadd-bce5e426d8bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7\" (UID: \"3662ca88-be02-42ef-aadd-bce5e426d8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:37:36.720175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.720141 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d2qb\" (UniqueName: \"kubernetes.io/projected/54f64bb1-139c-453e-9c86-d7289bbcbf44-kube-api-access-4d2qb\") pod \"payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4\" (UID: \"54f64bb1-139c-453e-9c86-d7289bbcbf44\") " pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:37:36.751435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:36.751400 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8pzc\" (UniqueName: \"kubernetes.io/projected/3662ca88-be02-42ef-aadd-bce5e426d8bb-kube-api-access-c8pzc\") pod \"payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7\" (UID: \"3662ca88-be02-42ef-aadd-bce5e426d8bb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:37:37.004735 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.004652 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:37:37.063132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.054776 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:37:37.434977 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:37:37.434949 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f64bb1_139c_453e_9c86_d7289bbcbf44.slice/crio-73dd54d9f157373027d8943c94ac24c70fb21d887b0853145dd0c78d25fedb6f WatchSource:0}: Error finding container 73dd54d9f157373027d8943c94ac24c70fb21d887b0853145dd0c78d25fedb6f: Status 404 returned error can't find the container with id 73dd54d9f157373027d8943c94ac24c70fb21d887b0853145dd0c78d25fedb6f Nov 01 22:37:37.436383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.436360 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4"] Nov 01 22:37:37.484972 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.484947 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7"] Nov 01 22:37:37.485087 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:37:37.485007 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3662ca88_be02_42ef_aadd_bce5e426d8bb.slice/crio-670dd0dbbe41adeb359f24ebbbbf1f4b9789e47147a20fe00494be2d43246438 WatchSource:0}: Error finding container 670dd0dbbe41adeb359f24ebbbbf1f4b9789e47147a20fe00494be2d43246438: Status 404 returned error can't find the container with id 670dd0dbbe41adeb359f24ebbbbf1f4b9789e47147a20fe00494be2d43246438 Nov 01 22:37:37.905234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.905197 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" event={"ID":"3662ca88-be02-42ef-aadd-bce5e426d8bb","Type":"ContainerStarted","Data":"f199c08ac85ca4c7bd3bfda753113a53f9207ca75c84277bd4ea2289ae11cc9c"} Nov 01 22:37:37.905464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.905275 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" event={"ID":"3662ca88-be02-42ef-aadd-bce5e426d8bb","Type":"ContainerStarted","Data":"670dd0dbbe41adeb359f24ebbbbf1f4b9789e47147a20fe00494be2d43246438"} Nov 01 22:37:37.905464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.905419 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:37:37.907420 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.907379 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" event={"ID":"54f64bb1-139c-453e-9c86-d7289bbcbf44","Type":"ContainerStarted","Data":"1000c76694aca1f4f7167cfa670b9fe21c064391822c47fe68c4f184df72a6cd"} Nov 01 22:37:37.907547 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.907444 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:37:37.907547 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.907462 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" event={"ID":"54f64bb1-139c-453e-9c86-d7289bbcbf44","Type":"ContainerStarted","Data":"73dd54d9f157373027d8943c94ac24c70fb21d887b0853145dd0c78d25fedb6f"} Nov 01 22:37:37.929811 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.929756 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" podStartSLOduration=1.92973991 podStartE2EDuration="1.92973991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:37.925371836 +0000 UTC m=+861.709810759" watchObservedRunningTime="2026-11-01 22:37:37.92973991 +0000 UTC m=+861.714178830" Nov 01 22:37:37.949948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:37.949883 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" podStartSLOduration=1.9498656909999998 podStartE2EDuration="1.949865691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:37.949095762 +0000 UTC m=+861.733534684" watchObservedRunningTime="2026-11-01 22:37:37.949865691 +0000 UTC m=+861.734304613" Nov 01 22:37:38.233855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:38.233764 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68"] Nov 01 22:37:38.260316 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:38.260221 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8"] Nov 01 22:37:40.782299 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:40.782161 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz"] Nov 01 22:37:40.782849 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:40.782502 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" podUID="cc5375cc-c59d-4614-8137-5782f114cb1f" containerName="maas-api" containerID="cri-o://00e7a0409e94663c8a732a08354cd18b8114d0e87e94436724123c781a7d7ca9" gracePeriod=30 Nov 01 22:37:41.068565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.068017 6466 generic.go:356] "Generic (PLEG): container finished" podID="cc5375cc-c59d-4614-8137-5782f114cb1f" containerID="00e7a0409e94663c8a732a08354cd18b8114d0e87e94436724123c781a7d7ca9" exitCode=0 Nov 01 22:37:41.068565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.068302 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" event={"ID":"cc5375cc-c59d-4614-8137-5782f114cb1f","Type":"ContainerDied","Data":"00e7a0409e94663c8a732a08354cd18b8114d0e87e94436724123c781a7d7ca9"} Nov 01 22:37:41.814760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.814731 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:41.909297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.909269 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc5375cc-c59d-4614-8137-5782f114cb1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc5375cc-c59d-4614-8137-5782f114cb1f-openshift-service-ca\") pod \"cc5375cc-c59d-4614-8137-5782f114cb1f\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " Nov 01 22:37:41.909452 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.909348 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-maas-api-tls\") pod \"cc5375cc-c59d-4614-8137-5782f114cb1f\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " Nov 01 22:37:41.909540 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.909476 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5375cc-c59d-4614-8137-5782f114cb1f-kube-api-access-g9wmj\" (UniqueName: \"kubernetes.io/projected/cc5375cc-c59d-4614-8137-5782f114cb1f-kube-api-access-g9wmj\") pod \"cc5375cc-c59d-4614-8137-5782f114cb1f\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " Nov 01 22:37:41.909605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.909591 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-metrics-certs\") pod \"cc5375cc-c59d-4614-8137-5782f114cb1f\" (UID: \"cc5375cc-c59d-4614-8137-5782f114cb1f\") " Nov 01 22:37:41.909716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.909687 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc5375cc-c59d-4614-8137-5782f114cb1f-openshift-service-ca" pod "cc5375cc-c59d-4614-8137-5782f114cb1f" (UID: "cc5375cc-c59d-4614-8137-5782f114cb1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:41.910422 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.910397 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc5375cc-c59d-4614-8137-5782f114cb1f-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:41.911690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.911665 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-metrics-certs" pod "cc5375cc-c59d-4614-8137-5782f114cb1f" (UID: "cc5375cc-c59d-4614-8137-5782f114cb1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:41.911921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.911896 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-maas-api-tls" pod "cc5375cc-c59d-4614-8137-5782f114cb1f" (UID: "cc5375cc-c59d-4614-8137-5782f114cb1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:41.911989 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:41.911935 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5375cc-c59d-4614-8137-5782f114cb1f-kube-api-access-g9wmj" pod "cc5375cc-c59d-4614-8137-5782f114cb1f" (UID: "cc5375cc-c59d-4614-8137-5782f114cb1f"). InnerVolumeSpecName "kube-api-access-g9wmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:42.012601 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.012507 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:42.012601 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.012543 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9wmj\" (UniqueName: \"kubernetes.io/projected/cc5375cc-c59d-4614-8137-5782f114cb1f-kube-api-access-g9wmj\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:42.012601 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.012558 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc5375cc-c59d-4614-8137-5782f114cb1f-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:42.078094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.078058 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" Nov 01 22:37:42.078094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.078083 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" event={"ID":"cc5375cc-c59d-4614-8137-5782f114cb1f","Type":"ContainerDied","Data":"6a76c82f4e129f6976531c94a26ed5c1c675a03b86323ba5e07d61fc45f7efe4"} Nov 01 22:37:42.078363 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.078163 6466 scope.go:122] "RemoveContainer" containerID="00e7a0409e94663c8a732a08354cd18b8114d0e87e94436724123c781a7d7ca9" Nov 01 22:37:42.102187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.102151 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz"] Nov 01 22:37:42.108414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.108389 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz"] Nov 01 22:37:42.395060 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.395028 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc5375cc-c59d-4614-8137-5782f114cb1f" path="/var/lib/kubelet/pods/cc5375cc-c59d-4614-8137-5782f114cb1f/volumes" Nov 01 22:37:42.507861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.507827 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57"] Nov 01 22:37:42.509002 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.508972 6466 cpu_manager.go:414] "containerMap: removing container" podUID="cc5375cc-c59d-4614-8137-5782f114cb1f" containerName="maas-api" Nov 01 22:37:42.509002 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.509001 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5375cc-c59d-4614-8137-5782f114cb1f" containerName="maas-api" Nov 01 22:37:42.509217 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.509181 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5375cc-c59d-4614-8137-5782f114cb1f" containerName="maas-api" Nov 01 22:37:42.515042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.515019 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.523516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.523464 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-0fd159-openshift-default-dockercfg-kncn2\"" Nov 01 22:37:42.536655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.534390 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57"] Nov 01 22:37:42.647194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.626935 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-certs\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.667134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.667077 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-socket\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.667467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.667436 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-data\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.667676 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.667656 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d8ca67a-a7ed-4a7c-905e-f02763779799-istiod-ca-cert\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.667887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.667870 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-podinfo\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.668101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.668085 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-credential-socket\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.668239 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.668223 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f52hf\" (UniqueName: \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-kube-api-access-f52hf\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.668442 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.668423 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-envoy\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.670165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.668618 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-token\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.773357 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.773320 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-token\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.777352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.777315 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-certs\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.806244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.806213 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-socket\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.806552 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.806529 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-data\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.806727 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.806711 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d8ca67a-a7ed-4a7c-905e-f02763779799-istiod-ca-cert\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.806919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.806905 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-podinfo\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.807139 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.807122 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-credential-socket\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.809178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.809153 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f52hf\" (UniqueName: \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-kube-api-access-f52hf\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.809440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.809420 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-envoy\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.809732 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.808235 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d8ca67a-a7ed-4a7c-905e-f02763779799-istiod-ca-cert\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.809838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.808851 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-credential-socket\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.809925 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.809081 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-data\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.810010 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.805561 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-certs\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.810383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.807625 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-socket\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.828139 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.824034 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-podinfo\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.829084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.829055 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-envoy\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.868133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.863928 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-token\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:42.868133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:42.864956 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f52hf\" (UniqueName: \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-kube-api-access-f52hf\") pod \"e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:43.147254 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.147223 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:43.239313 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.234967 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" podUID="7deebd9c-6143-45e1-934b-12bdf6cd416f" containerName="payload-processing" containerID="cri-o://d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3" gracePeriod=25 Nov 01 22:37:43.239313 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.235305 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9"] Nov 01 22:37:43.265448 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.254409 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.265448 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.261536 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" podUID="e867a1df-a0ee-496b-a331-3d6ade14bee2" containerName="payload-pre-processing" containerID="cri-o://f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1" gracePeriod=25 Nov 01 22:37:43.271332 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.271298 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9"] Nov 01 22:37:43.336508 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.336485 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-66033bd2-serving-cert\"" Nov 01 22:37:43.402567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.402478 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztfhx\" (UniqueName: \"kubernetes.io/projected/d757b14b-037a-429b-a537-268485243679-kube-api-access-ztfhx\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.402859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.402835 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d757b14b-037a-429b-a537-268485243679-openshift-service-ca\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.403248 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.403229 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-metrics-certs\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.403406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.403391 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-maas-api-tls\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.514003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.513965 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-metrics-certs\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.514782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.514755 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-maas-api-tls\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.519468 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.516541 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfhx\" (UniqueName: \"kubernetes.io/projected/d757b14b-037a-429b-a537-268485243679-kube-api-access-ztfhx\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.519468 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.516617 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d757b14b-037a-429b-a537-268485243679-openshift-service-ca\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.519468 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.517845 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d757b14b-037a-429b-a537-268485243679-openshift-service-ca\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.530379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.529266 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57"] Nov 01 22:37:43.535678 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.535651 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-maas-api-tls\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.542011 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.539696 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztfhx\" (UniqueName: \"kubernetes.io/projected/d757b14b-037a-429b-a537-268485243679-kube-api-access-ztfhx\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.552329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.549370 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-metrics-certs\") pod \"maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.553279 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.553247 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:43.553389 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.553316 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:43.553389 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.553352 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:43.583491 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.583464 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:43.736867 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.736837 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:43.895778 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.895749 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:43.953265 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.948089 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7deebd9c-6143-45e1-934b-12bdf6cd416f-kube-api-access-sgrqf\" (UniqueName: \"kubernetes.io/projected/7deebd9c-6143-45e1-934b-12bdf6cd416f-kube-api-access-sgrqf\") pod \"7deebd9c-6143-45e1-934b-12bdf6cd416f\" (UID: \"7deebd9c-6143-45e1-934b-12bdf6cd416f\") " Nov 01 22:37:43.953265 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.948179 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7deebd9c-6143-45e1-934b-12bdf6cd416f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7deebd9c-6143-45e1-934b-12bdf6cd416f-plugins-config-volume\") pod \"7deebd9c-6143-45e1-934b-12bdf6cd416f\" (UID: \"7deebd9c-6143-45e1-934b-12bdf6cd416f\") " Nov 01 22:37:43.998188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:43.994391 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7deebd9c-6143-45e1-934b-12bdf6cd416f-kube-api-access-sgrqf" pod "7deebd9c-6143-45e1-934b-12bdf6cd416f" (UID: "7deebd9c-6143-45e1-934b-12bdf6cd416f"). InnerVolumeSpecName "kube-api-access-sgrqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:44.019214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.013182 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7deebd9c-6143-45e1-934b-12bdf6cd416f-plugins-config-volume" pod "7deebd9c-6143-45e1-934b-12bdf6cd416f" (UID: "7deebd9c-6143-45e1-934b-12bdf6cd416f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:44.115419 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.051648 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e867a1df-a0ee-496b-a331-3d6ade14bee2-kube-api-access-q8rfb\" (UniqueName: \"kubernetes.io/projected/e867a1df-a0ee-496b-a331-3d6ade14bee2-kube-api-access-q8rfb\") pod \"e867a1df-a0ee-496b-a331-3d6ade14bee2\" (UID: \"e867a1df-a0ee-496b-a331-3d6ade14bee2\") " Nov 01 22:37:44.115419 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.051971 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e867a1df-a0ee-496b-a331-3d6ade14bee2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e867a1df-a0ee-496b-a331-3d6ade14bee2-plugins-config-volume\") pod \"e867a1df-a0ee-496b-a331-3d6ade14bee2\" (UID: \"e867a1df-a0ee-496b-a331-3d6ade14bee2\") " Nov 01 22:37:44.115419 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.052710 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgrqf\" (UniqueName: \"kubernetes.io/projected/7deebd9c-6143-45e1-934b-12bdf6cd416f-kube-api-access-sgrqf\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:44.115419 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.052737 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7deebd9c-6143-45e1-934b-12bdf6cd416f-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:44.115419 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.099338 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e867a1df-a0ee-496b-a331-3d6ade14bee2-kube-api-access-q8rfb" pod "e867a1df-a0ee-496b-a331-3d6ade14bee2" (UID: "e867a1df-a0ee-496b-a331-3d6ade14bee2"). InnerVolumeSpecName "kube-api-access-q8rfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:44.115419 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.108569 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e867a1df-a0ee-496b-a331-3d6ade14bee2-plugins-config-volume" pod "e867a1df-a0ee-496b-a331-3d6ade14bee2" (UID: "e867a1df-a0ee-496b-a331-3d6ade14bee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:44.116852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.116805 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9"] Nov 01 22:37:44.147384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.147346 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" event={"ID":"7d8ca67a-a7ed-4a7c-905e-f02763779799","Type":"ContainerStarted","Data":"5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366"} Nov 01 22:37:44.147655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.147626 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" event={"ID":"7d8ca67a-a7ed-4a7c-905e-f02763779799","Type":"ContainerStarted","Data":"1ecb87ed4185d436f2d3a290f6b680aef39759fdb8aadad1acf45b1784d8c4bd"} Nov 01 22:37:44.153539 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.153513 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:44.155551 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.155415 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e867a1df-a0ee-496b-a331-3d6ade14bee2-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:44.155768 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.155751 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8rfb\" (UniqueName: \"kubernetes.io/projected/e867a1df-a0ee-496b-a331-3d6ade14bee2-kube-api-access-q8rfb\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:44.158894 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.155970 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 22:37:44.159127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.159086 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 22:37:44.161832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.161801 6466 generic.go:356] "Generic (PLEG): container finished" podID="e867a1df-a0ee-496b-a331-3d6ade14bee2" containerID="f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1" exitCode=0 Nov 01 22:37:44.162163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.162137 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" event={"ID":"e867a1df-a0ee-496b-a331-3d6ade14bee2","Type":"ContainerDied","Data":"f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1"} Nov 01 22:37:44.162336 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.162319 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" event={"ID":"e867a1df-a0ee-496b-a331-3d6ade14bee2","Type":"ContainerDied","Data":"ef1b7617da159c5b3278ad344727e8ab64f9a513b067921dc9349ec8135142a5"} Nov 01 22:37:44.162447 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.162435 6466 scope.go:122] "RemoveContainer" containerID="f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1" Nov 01 22:37:44.162706 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.162690 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8" Nov 01 22:37:44.169464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.169440 6466 generic.go:356] "Generic (PLEG): container finished" podID="7deebd9c-6143-45e1-934b-12bdf6cd416f" containerID="d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3" exitCode=0 Nov 01 22:37:44.169760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.169713 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" event={"ID":"7deebd9c-6143-45e1-934b-12bdf6cd416f","Type":"ContainerDied","Data":"d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3"} Nov 01 22:37:44.169948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.169921 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" event={"ID":"7deebd9c-6143-45e1-934b-12bdf6cd416f","Type":"ContainerDied","Data":"45d4d7825095982800a458bc63b73364793c7665512528a0a6ff180a47880cc6"} Nov 01 22:37:44.170175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.170161 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68" Nov 01 22:37:44.296529 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.296475 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podStartSLOduration=2.296452731 podStartE2EDuration="2.296452731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:44.171304377 +0000 UTC m=+867.955743294" watchObservedRunningTime="2026-11-01 22:37:44.296452731 +0000 UTC m=+868.080891653" Nov 01 22:37:44.304445 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.303253 6466 scope.go:122] "RemoveContainer" containerID="f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1" Nov 01 22:37:44.305020 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:44.304994 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1\": container with ID starting with f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1 not found: ID does not exist" containerID="f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1" Nov 01 22:37:44.305214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.305185 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1"} err="failed to get container status \"f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1\": rpc error: code = NotFound desc = could not find container \"f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1\": container with ID starting with f0d3a532040d6e6f580cca7a316591ee18093f8abf149126093c87a8eba498a1 not found: ID does not exist" Nov 01 22:37:44.305343 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.305332 6466 scope.go:122] "RemoveContainer" containerID="d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3" Nov 01 22:37:44.322253 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.322226 6466 scope.go:122] "RemoveContainer" containerID="d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3" Nov 01 22:37:44.322604 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:44.322583 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3\": container with ID starting with d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3 not found: ID does not exist" containerID="d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3" Nov 01 22:37:44.322676 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.322613 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3"} err="failed to get container status \"d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3\": rpc error: code = NotFound desc = could not find container \"d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3\": container with ID starting with d877c9db6a0222f698711e96e8f1da21118b077f3ab660c3649bf71c234890f3 not found: ID does not exist" Nov 01 22:37:44.333148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.332283 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8"] Nov 01 22:37:44.340282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.340230 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-61ce6e07-66dddf58dc-jf8b8"] Nov 01 22:37:44.427133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.424498 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e867a1df-a0ee-496b-a331-3d6ade14bee2" path="/var/lib/kubelet/pods/e867a1df-a0ee-496b-a331-3d6ade14bee2/volumes" Nov 01 22:37:44.427133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.425193 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68"] Nov 01 22:37:44.427133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.425237 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-61ce6e07-79695f4d99-nsq68"] Nov 01 22:37:44.601858 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:44.601825 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ce6e07-66dc7b8979-zgrnz" podUID="cc5375cc-c59d-4614-8137-5782f114cb1f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.93:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:45.153276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:45.153246 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:45.181358 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:45.181323 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" event={"ID":"d757b14b-037a-429b-a537-268485243679","Type":"ContainerStarted","Data":"5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2"} Nov 01 22:37:45.181533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:45.181380 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" event={"ID":"d757b14b-037a-429b-a537-268485243679","Type":"ContainerStarted","Data":"87f9dda7a33c54f38180febd6ab75e38553285fa4e4c8de48eaf071b40d302cf"} Nov 01 22:37:45.181533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:45.181423 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:45.182730 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:45.182706 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:45.183553 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:45.183537 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:37:45.204449 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:45.204405 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" podStartSLOduration=1.753706755 podStartE2EDuration="2.204392198s" totalImagesPullingTime="450.685443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:45.200898817 +0000 UTC m=+868.985337740" watchObservedRunningTime="2026-11-01 22:37:45.204392198 +0000 UTC m=+868.988831145" Nov 01 22:37:46.346959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:46.346919 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7deebd9c-6143-45e1-934b-12bdf6cd416f" path="/var/lib/kubelet/pods/7deebd9c-6143-45e1-934b-12bdf6cd416f/volumes" Nov 01 22:37:47.608924 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.608888 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn"] Nov 01 22:37:47.609703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.609687 6466 cpu_manager.go:414] "containerMap: removing container" podUID="7deebd9c-6143-45e1-934b-12bdf6cd416f" containerName="payload-processing" Nov 01 22:37:47.609758 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.609705 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7deebd9c-6143-45e1-934b-12bdf6cd416f" containerName="payload-processing" Nov 01 22:37:47.609758 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.609737 6466 cpu_manager.go:414] "containerMap: removing container" podUID="e867a1df-a0ee-496b-a331-3d6ade14bee2" containerName="payload-pre-processing" Nov 01 22:37:47.609758 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.609742 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e867a1df-a0ee-496b-a331-3d6ade14bee2" containerName="payload-pre-processing" Nov 01 22:37:47.609862 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.609853 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e867a1df-a0ee-496b-a331-3d6ade14bee2" containerName="payload-pre-processing" Nov 01 22:37:47.609897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.609870 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7deebd9c-6143-45e1-934b-12bdf6cd416f" containerName="payload-processing" Nov 01 22:37:47.615017 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.614994 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:37:47.620476 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.620452 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-0fd159\"" Nov 01 22:37:47.622075 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.621983 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn"] Nov 01 22:37:47.646773 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.646736 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qdzm\" (UniqueName: \"kubernetes.io/projected/69034553-3a33-4319-b857-6fed9c037c1a-kube-api-access-5qdzm\") pod \"payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn\" (UID: \"69034553-3a33-4319-b857-6fed9c037c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:37:47.646998 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.646981 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69034553-3a33-4319-b857-6fed9c037c1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn\" (UID: \"69034553-3a33-4319-b857-6fed9c037c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:37:47.772129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.762793 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69034553-3a33-4319-b857-6fed9c037c1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn\" (UID: \"69034553-3a33-4319-b857-6fed9c037c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:37:47.772129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.763000 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdzm\" (UniqueName: \"kubernetes.io/projected/69034553-3a33-4319-b857-6fed9c037c1a-kube-api-access-5qdzm\") pod \"payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn\" (UID: \"69034553-3a33-4319-b857-6fed9c037c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:37:47.772129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.764313 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69034553-3a33-4319-b857-6fed9c037c1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn\" (UID: \"69034553-3a33-4319-b857-6fed9c037c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:37:47.953135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:47.937867 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qdzm\" (UniqueName: \"kubernetes.io/projected/69034553-3a33-4319-b857-6fed9c037c1a-kube-api-access-5qdzm\") pod \"payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn\" (UID: \"69034553-3a33-4319-b857-6fed9c037c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:37:48.241187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:48.239097 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm"] Nov 01 22:37:48.241187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:48.240440 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:37:48.409214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:48.407332 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6"] Nov 01 22:37:48.964140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:48.962202 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz"] Nov 01 22:37:48.964140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:48.962510 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" podUID="629d4954-f71d-4354-b29c-d660b2fc922b" containerName="maas-api" containerID="cri-o://c3fc83c1b8311084ca2b00f79095b78374d9e3b5f51a468827802700042c16b3" gracePeriod=30 Nov 01 22:37:48.986527 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:48.977338 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:37:49.019580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.008789 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:37:49.085528 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.083368 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn"] Nov 01 22:37:49.471667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.471616 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" event={"ID":"69034553-3a33-4319-b857-6fed9c037c1a","Type":"ContainerStarted","Data":"b13e3ad5b8d999306d745192b094a06e84808e4ba235051613055c37bf176adc"} Nov 01 22:37:49.474012 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.473982 6466 generic.go:356] "Generic (PLEG): container finished" podID="629d4954-f71d-4354-b29c-d660b2fc922b" containerID="c3fc83c1b8311084ca2b00f79095b78374d9e3b5f51a468827802700042c16b3" exitCode=0 Nov 01 22:37:49.474585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.474557 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" event={"ID":"629d4954-f71d-4354-b29c-d660b2fc922b","Type":"ContainerDied","Data":"c3fc83c1b8311084ca2b00f79095b78374d9e3b5f51a468827802700042c16b3"} Nov 01 22:37:49.574973 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:49.574923 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:37:49.795356 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.795328 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:49.882939 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.882900 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629d4954-f71d-4354-b29c-d660b2fc922b-kube-api-access-7gk9v\" (UniqueName: \"kubernetes.io/projected/629d4954-f71d-4354-b29c-d660b2fc922b-kube-api-access-7gk9v\") pod \"629d4954-f71d-4354-b29c-d660b2fc922b\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " Nov 01 22:37:49.883122 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.883009 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-maas-api-tls\") pod \"629d4954-f71d-4354-b29c-d660b2fc922b\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " Nov 01 22:37:49.883191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.883142 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-metrics-certs\") pod \"629d4954-f71d-4354-b29c-d660b2fc922b\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " Nov 01 22:37:49.884342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.883703 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629d4954-f71d-4354-b29c-d660b2fc922b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/629d4954-f71d-4354-b29c-d660b2fc922b-openshift-service-ca\") pod \"629d4954-f71d-4354-b29c-d660b2fc922b\" (UID: \"629d4954-f71d-4354-b29c-d660b2fc922b\") " Nov 01 22:37:49.884342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.884308 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629d4954-f71d-4354-b29c-d660b2fc922b-openshift-service-ca" pod "629d4954-f71d-4354-b29c-d660b2fc922b" (UID: "629d4954-f71d-4354-b29c-d660b2fc922b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:49.884851 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.884828 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/629d4954-f71d-4354-b29c-d660b2fc922b-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:49.888553 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.888517 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-metrics-certs" pod "629d4954-f71d-4354-b29c-d660b2fc922b" (UID: "629d4954-f71d-4354-b29c-d660b2fc922b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:49.889404 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.889314 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629d4954-f71d-4354-b29c-d660b2fc922b-kube-api-access-7gk9v" pod "629d4954-f71d-4354-b29c-d660b2fc922b" (UID: "629d4954-f71d-4354-b29c-d660b2fc922b"). InnerVolumeSpecName "kube-api-access-7gk9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:49.889404 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.889371 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-maas-api-tls" pod "629d4954-f71d-4354-b29c-d660b2fc922b" (UID: "629d4954-f71d-4354-b29c-d660b2fc922b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:49.893666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.893641 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv"] Nov 01 22:37:49.893906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:49.893882 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" podUID="fb3ce252-0ab4-499d-abac-fed6118d1273" containerName="istio-proxy" containerID="cri-o://613431851b55a7d24b7dfb75fe3c143aa5489b1a62feea91e40fb1a5d8633d44" gracePeriod=30 Nov 01 22:37:50.018184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:50.004788 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gk9v\" (UniqueName: \"kubernetes.io/projected/629d4954-f71d-4354-b29c-d660b2fc922b-kube-api-access-7gk9v\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:50.018184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:50.004832 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:50.018184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:50.004848 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629d4954-f71d-4354-b29c-d660b2fc922b-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:50.538729 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:50.538687 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" event={"ID":"69034553-3a33-4319-b857-6fed9c037c1a","Type":"ContainerStarted","Data":"c6c091869d2ba4563d6f4602e059b88a3e7564b97b47cdc1a1aae1ef6266533d"} Nov 01 22:37:50.540461 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:50.540422 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:37:50.543050 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:50.543020 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" event={"ID":"629d4954-f71d-4354-b29c-d660b2fc922b","Type":"ContainerDied","Data":"580ce90bbd612ddc00ddcf76828d20091f24aa30e5570fe78e351d17c86f3836"} Nov 01 22:37:50.543290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:50.543271 6466 scope.go:122] "RemoveContainer" containerID="c3fc83c1b8311084ca2b00f79095b78374d9e3b5f51a468827802700042c16b3" Nov 01 22:37:50.544134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:50.543695 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" Nov 01 22:37:50.594008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:50.593959 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" podStartSLOduration=3.593944395 podStartE2EDuration="3.593944395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:50.587642779 +0000 UTC m=+874.372081701" watchObservedRunningTime="2026-11-01 22:37:50.593944395 +0000 UTC m=+874.378383315" Nov 01 22:37:50.622141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:50.620800 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz"] Nov 01 22:37:50.625702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:50.625671 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz"] Nov 01 22:37:51.048734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.048688 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9"] Nov 01 22:37:51.050731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.050688 6466 cpu_manager.go:414] "containerMap: removing container" podUID="629d4954-f71d-4354-b29c-d660b2fc922b" containerName="maas-api" Nov 01 22:37:51.050731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.050714 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629d4954-f71d-4354-b29c-d660b2fc922b" containerName="maas-api" Nov 01 22:37:51.065075 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.064726 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="629d4954-f71d-4354-b29c-d660b2fc922b" containerName="maas-api" Nov 01 22:37:51.077627 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.077600 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.081182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.081152 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9"] Nov 01 22:37:51.082296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.082274 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-10a1d9-openshift-default-dockercfg-9mgc9\"" Nov 01 22:37:51.187136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.182389 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-data\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.187136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.182468 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-socket\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.187136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.182555 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-podinfo\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.187136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.182579 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-certs\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.187136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.182685 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t722x\" (UniqueName: \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-kube-api-access-t722x\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.187136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.182724 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1280b1-f9ea-489c-8b06-f595fae26903-istiod-ca-cert\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.187136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.182789 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-token\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.187136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.182951 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-credential-socket\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.187136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.182974 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-envoy\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.267351 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.267321 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:37:51.295627 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.295588 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-token\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.296377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.296348 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-credential-socket\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.296575 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.296554 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-envoy\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.296906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.296887 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-data\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.297040 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.297027 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-socket\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.297220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.297206 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-podinfo\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.297319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.297304 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-certs\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.297524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.297508 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t722x\" (UniqueName: \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-kube-api-access-t722x\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.317567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.317499 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1280b1-f9ea-489c-8b06-f595fae26903-istiod-ca-cert\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.338127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.328520 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-token\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.338408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.335155 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-socket\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.338555 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.316578 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-data\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.338667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.316834 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-credential-socket\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.338755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.336583 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1280b1-f9ea-489c-8b06-f595fae26903-istiod-ca-cert\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.359183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.351331 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-envoy\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.370129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.359747 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-certs\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.370129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.369077 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-podinfo\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.378720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.378649 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t722x\" (UniqueName: \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-kube-api-access-t722x\") pod \"e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:51.413141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:51.400433 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:52.629900 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:37:52.629859 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1280b1_f9ea_489c_8b06_f595fae26903.slice/crio-9462b93b3b1ad5e3b611a3eb579647f50493ebeaa905cc1d7d1dc9b57b802db5 WatchSource:0}: Error finding container 9462b93b3b1ad5e3b611a3eb579647f50493ebeaa905cc1d7d1dc9b57b802db5: Status 404 returned error can't find the container with id 9462b93b3b1ad5e3b611a3eb579647f50493ebeaa905cc1d7d1dc9b57b802db5 Nov 01 22:37:52.637120 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:52.634189 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:52.637120 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:52.634283 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:52.637120 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:52.634324 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:37:52.642895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:52.642872 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629d4954-f71d-4354-b29c-d660b2fc922b" path="/var/lib/kubelet/pods/629d4954-f71d-4354-b29c-d660b2fc922b/volumes" Nov 01 22:37:52.643436 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:52.643416 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9"] Nov 01 22:37:52.724347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:52.724300 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e4b83ffd-5498577445-qqthz" podUID="629d4954-f71d-4354-b29c-d660b2fc922b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.96:8443/health\": context deadline exceeded" Nov 01 22:37:52.878387 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:52.874950 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" event={"ID":"1f1280b1-f9ea-489c-8b06-f595fae26903","Type":"ContainerStarted","Data":"9462b93b3b1ad5e3b611a3eb579647f50493ebeaa905cc1d7d1dc9b57b802db5"} Nov 01 22:37:53.239621 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:53.239561 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" podUID="0321f7c9-74fa-470a-87e5-88e1f9b21ca1" containerName="payload-processing" containerID="cri-o://ba86b646d37a67ac6a49e8472e005bc71fe526464a8dfe3ce938c19a3105ed1d" gracePeriod=25 Nov 01 22:37:53.408428 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:53.408362 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" podUID="19aa924b-c7f5-4622-9c6f-865fc6e33897" containerName="payload-pre-processing" containerID="cri-o://d9a6e26d15508ac2a15f913b415b00104bfffca5b099fc8894d46b6c16869dd6" gracePeriod=25 Nov 01 22:37:54.018821 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.018790 6466 generic.go:356] "Generic (PLEG): container finished" podID="0321f7c9-74fa-470a-87e5-88e1f9b21ca1" containerID="ba86b646d37a67ac6a49e8472e005bc71fe526464a8dfe3ce938c19a3105ed1d" exitCode=0 Nov 01 22:37:54.019406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.019380 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" event={"ID":"0321f7c9-74fa-470a-87e5-88e1f9b21ca1","Type":"ContainerDied","Data":"ba86b646d37a67ac6a49e8472e005bc71fe526464a8dfe3ce938c19a3105ed1d"} Nov 01 22:37:54.022728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.022682 6466 generic.go:356] "Generic (PLEG): container finished" podID="19aa924b-c7f5-4622-9c6f-865fc6e33897" containerID="d9a6e26d15508ac2a15f913b415b00104bfffca5b099fc8894d46b6c16869dd6" exitCode=0 Nov 01 22:37:54.022964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.022943 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" event={"ID":"19aa924b-c7f5-4622-9c6f-865fc6e33897","Type":"ContainerDied","Data":"d9a6e26d15508ac2a15f913b415b00104bfffca5b099fc8894d46b6c16869dd6"} Nov 01 22:37:54.024992 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.024964 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" event={"ID":"1f1280b1-f9ea-489c-8b06-f595fae26903","Type":"ContainerStarted","Data":"a0e350bd90b75c131e38853e602f0a1f838db8a93983274d3106b3544cf9c217"} Nov 01 22:37:54.056003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.055969 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:54.058935 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.058432 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podStartSLOduration=3.058417468 podStartE2EDuration="3.058417468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:54.054768061 +0000 UTC m=+877.839206982" watchObservedRunningTime="2026-11-01 22:37:54.058417468 +0000 UTC m=+877.842856389" Nov 01 22:37:54.146675 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.146637 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-kube-api-access-l9gs2\" (UniqueName: \"kubernetes.io/projected/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-kube-api-access-l9gs2\") pod \"0321f7c9-74fa-470a-87e5-88e1f9b21ca1\" (UID: \"0321f7c9-74fa-470a-87e5-88e1f9b21ca1\") " Nov 01 22:37:54.146912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.146740 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-plugins-config-volume\") pod \"0321f7c9-74fa-470a-87e5-88e1f9b21ca1\" (UID: \"0321f7c9-74fa-470a-87e5-88e1f9b21ca1\") " Nov 01 22:37:54.147221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.147188 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-plugins-config-volume" pod "0321f7c9-74fa-470a-87e5-88e1f9b21ca1" (UID: "0321f7c9-74fa-470a-87e5-88e1f9b21ca1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:54.147490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.147471 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:54.153595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.153568 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-kube-api-access-l9gs2" pod "0321f7c9-74fa-470a-87e5-88e1f9b21ca1" (UID: "0321f7c9-74fa-470a-87e5-88e1f9b21ca1"). InnerVolumeSpecName "kube-api-access-l9gs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:54.270553 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.266688 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:54.338808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.337800 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9gs2\" (UniqueName: \"kubernetes.io/projected/0321f7c9-74fa-470a-87e5-88e1f9b21ca1-kube-api-access-l9gs2\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:54.405180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.405149 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:54.412510 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.412482 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:54.438463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.438429 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19aa924b-c7f5-4622-9c6f-865fc6e33897-kube-api-access-6nhcq\" (UniqueName: \"kubernetes.io/projected/19aa924b-c7f5-4622-9c6f-865fc6e33897-kube-api-access-6nhcq\") pod \"19aa924b-c7f5-4622-9c6f-865fc6e33897\" (UID: \"19aa924b-c7f5-4622-9c6f-865fc6e33897\") " Nov 01 22:37:54.438638 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.438567 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19aa924b-c7f5-4622-9c6f-865fc6e33897-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19aa924b-c7f5-4622-9c6f-865fc6e33897-plugins-config-volume\") pod \"19aa924b-c7f5-4622-9c6f-865fc6e33897\" (UID: \"19aa924b-c7f5-4622-9c6f-865fc6e33897\") " Nov 01 22:37:54.438946 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.438921 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19aa924b-c7f5-4622-9c6f-865fc6e33897-plugins-config-volume" pod "19aa924b-c7f5-4622-9c6f-865fc6e33897" (UID: "19aa924b-c7f5-4622-9c6f-865fc6e33897"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:54.439828 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.439802 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19aa924b-c7f5-4622-9c6f-865fc6e33897-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:54.441378 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.441353 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19aa924b-c7f5-4622-9c6f-865fc6e33897-kube-api-access-6nhcq" pod "19aa924b-c7f5-4622-9c6f-865fc6e33897" (UID: "19aa924b-c7f5-4622-9c6f-865fc6e33897"). InnerVolumeSpecName "kube-api-access-6nhcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:54.466561 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:37:54.466539 6466 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0321f7c9_74fa_470a_87e5_88e1f9b21ca1.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:37:54.541631 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:54.541541 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhcq\" (UniqueName: \"kubernetes.io/projected/19aa924b-c7f5-4622-9c6f-865fc6e33897-kube-api-access-6nhcq\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:55.034956 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.034924 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" Nov 01 22:37:55.035570 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.035542 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm" event={"ID":"0321f7c9-74fa-470a-87e5-88e1f9b21ca1","Type":"ContainerDied","Data":"e4befdb3731e62ab061ac6a95d1c6afbfc4268f84422df22bdb6fd6858846eed"} Nov 01 22:37:55.035697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.035610 6466 scope.go:122] "RemoveContainer" containerID="ba86b646d37a67ac6a49e8472e005bc71fe526464a8dfe3ce938c19a3105ed1d" Nov 01 22:37:55.038649 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.038623 6466 generic.go:356] "Generic (PLEG): container finished" podID="fb3ce252-0ab4-499d-abac-fed6118d1273" containerID="613431851b55a7d24b7dfb75fe3c143aa5489b1a62feea91e40fb1a5d8633d44" exitCode=0 Nov 01 22:37:55.038835 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.038689 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" event={"ID":"fb3ce252-0ab4-499d-abac-fed6118d1273","Type":"ContainerDied","Data":"613431851b55a7d24b7dfb75fe3c143aa5489b1a62feea91e40fb1a5d8633d44"} Nov 01 22:37:55.040721 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.040704 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" Nov 01 22:37:55.040843 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.040722 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6" event={"ID":"19aa924b-c7f5-4622-9c6f-865fc6e33897","Type":"ContainerDied","Data":"5abdaef52c22e023c7ae51f89c497ba9503f2ad616ba11fddabe709a432dcb30"} Nov 01 22:37:55.041088 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.041064 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:55.042344 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.042325 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:37:55.050072 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.049975 6466 scope.go:122] "RemoveContainer" containerID="d9a6e26d15508ac2a15f913b415b00104bfffca5b099fc8894d46b6c16869dd6" Nov 01 22:37:55.056462 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.056436 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm"] Nov 01 22:37:55.061453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.061431 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4b83ffd-56775bf959-ng5bm"] Nov 01 22:37:55.106341 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.106307 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6"] Nov 01 22:37:55.112547 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.112516 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4b83ffd-fcb56c5ff-72mv6"] Nov 01 22:37:55.273152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.271477 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:55.312668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.309076 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t"] Nov 01 22:37:55.312668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.310208 6466 cpu_manager.go:414] "containerMap: removing container" podUID="0321f7c9-74fa-470a-87e5-88e1f9b21ca1" containerName="payload-processing" Nov 01 22:37:55.312668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.310228 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0321f7c9-74fa-470a-87e5-88e1f9b21ca1" containerName="payload-processing" Nov 01 22:37:55.312668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.310292 6466 cpu_manager.go:414] "containerMap: removing container" podUID="fb3ce252-0ab4-499d-abac-fed6118d1273" containerName="istio-proxy" Nov 01 22:37:55.312668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.310300 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb3ce252-0ab4-499d-abac-fed6118d1273" containerName="istio-proxy" Nov 01 22:37:55.312668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.310311 6466 cpu_manager.go:414] "containerMap: removing container" podUID="19aa924b-c7f5-4622-9c6f-865fc6e33897" containerName="payload-pre-processing" Nov 01 22:37:55.312668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.310321 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19aa924b-c7f5-4622-9c6f-865fc6e33897" containerName="payload-pre-processing" Nov 01 22:37:55.312668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.310510 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="0321f7c9-74fa-470a-87e5-88e1f9b21ca1" containerName="payload-processing" Nov 01 22:37:55.312668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.310532 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb3ce252-0ab4-499d-abac-fed6118d1273" containerName="istio-proxy" Nov 01 22:37:55.312668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.310543 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="19aa924b-c7f5-4622-9c6f-865fc6e33897" containerName="payload-pre-processing" Nov 01 22:37:55.315393 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.315368 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.381542 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.381503 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-0fd159-serving-cert\"" Nov 01 22:37:55.388236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.385058 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t"] Nov 01 22:37:55.392311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.392286 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-token\" (UniqueName: \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-token\") pod \"fb3ce252-0ab4-499d-abac-fed6118d1273\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " Nov 01 22:37:55.393432 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.393383 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3ce252-0ab4-499d-abac-fed6118d1273-istiod-ca-cert" pod "fb3ce252-0ab4-499d-abac-fed6118d1273" (UID: "fb3ce252-0ab4-499d-abac-fed6118d1273"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:55.393594 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.393001 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb3ce252-0ab4-499d-abac-fed6118d1273-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3ce252-0ab4-499d-abac-fed6118d1273-istiod-ca-cert\") pod \"fb3ce252-0ab4-499d-abac-fed6118d1273\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " Nov 01 22:37:55.393723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.393708 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-socket\") pod \"fb3ce252-0ab4-499d-abac-fed6118d1273\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " Nov 01 22:37:55.394157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.394138 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-envoy\") pod \"fb3ce252-0ab4-499d-abac-fed6118d1273\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " Nov 01 22:37:55.394557 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.394540 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-podinfo\") pod \"fb3ce252-0ab4-499d-abac-fed6118d1273\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " Nov 01 22:37:55.395097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.395080 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-kube-api-access-rknv2\" (UniqueName: \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-kube-api-access-rknv2\") pod \"fb3ce252-0ab4-499d-abac-fed6118d1273\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " Nov 01 22:37:55.395532 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.395514 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-data\") pod \"fb3ce252-0ab4-499d-abac-fed6118d1273\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " Nov 01 22:37:55.396166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.396148 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-credential-socket\") pod \"fb3ce252-0ab4-499d-abac-fed6118d1273\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " Nov 01 22:37:55.403750 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.403722 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-certs\") pod \"fb3ce252-0ab4-499d-abac-fed6118d1273\" (UID: \"fb3ce252-0ab4-499d-abac-fed6118d1273\") " Nov 01 22:37:55.404514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.404489 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqrl2\" (UniqueName: \"kubernetes.io/projected/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-kube-api-access-cqrl2\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.404741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.394022 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-socket" pod "fb3ce252-0ab4-499d-abac-fed6118d1273" (UID: "fb3ce252-0ab4-499d-abac-fed6118d1273"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:55.404831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.404736 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-token" pod "fb3ce252-0ab4-499d-abac-fed6118d1273" (UID: "fb3ce252-0ab4-499d-abac-fed6118d1273"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:55.404831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.395875 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-data" pod "fb3ce252-0ab4-499d-abac-fed6118d1273" (UID: "fb3ce252-0ab4-499d-abac-fed6118d1273"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:55.404831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.396568 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-credential-socket" pod "fb3ce252-0ab4-499d-abac-fed6118d1273" (UID: "fb3ce252-0ab4-499d-abac-fed6118d1273"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:55.404831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.404056 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-certs" pod "fb3ce252-0ab4-499d-abac-fed6118d1273" (UID: "fb3ce252-0ab4-499d-abac-fed6118d1273"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:55.405094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405052 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-kube-api-access-rknv2" pod "fb3ce252-0ab4-499d-abac-fed6118d1273" (UID: "fb3ce252-0ab4-499d-abac-fed6118d1273"). InnerVolumeSpecName "kube-api-access-rknv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:55.405209 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405155 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-envoy" pod "fb3ce252-0ab4-499d-abac-fed6118d1273" (UID: "fb3ce252-0ab4-499d-abac-fed6118d1273"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:55.405278 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405245 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.405320 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405284 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.405367 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405321 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-podinfo" pod "fb3ce252-0ab4-499d-abac-fed6118d1273" (UID: "fb3ce252-0ab4-499d-abac-fed6118d1273"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:37:55.405442 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405410 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.405818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405724 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3ce252-0ab4-499d-abac-fed6118d1273-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:55.405818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405742 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:55.405818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405752 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:55.405818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405761 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:55.405818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405769 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rknv2\" (UniqueName: \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-kube-api-access-rknv2\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:55.405818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405778 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:55.405818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405786 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:55.405818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405794 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce252-0ab4-499d-abac-fed6118d1273-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:55.405818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.405802 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3ce252-0ab4-499d-abac-fed6118d1273-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:55.506728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.506697 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.506728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.506733 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.506934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.506777 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.506977 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.506935 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqrl2\" (UniqueName: \"kubernetes.io/projected/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-kube-api-access-cqrl2\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.507555 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.507530 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.509235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.509215 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.509326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.509300 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.526157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.526132 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqrl2\" (UniqueName: \"kubernetes.io/projected/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-kube-api-access-cqrl2\") pod \"maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:55.728196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:55.722982 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:56.052856 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:56.052762 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" event={"ID":"fb3ce252-0ab4-499d-abac-fed6118d1273","Type":"ContainerDied","Data":"df7578264be325597b4842c5ee8e7803e604c74077a5a3658e9ea386353c0eb8"} Nov 01 22:37:56.053463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:56.053443 6466 scope.go:122] "RemoveContainer" containerID="613431851b55a7d24b7dfb75fe3c143aa5489b1a62feea91e40fb1a5d8633d44" Nov 01 22:37:56.053578 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:56.052817 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv" Nov 01 22:37:56.061271 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:37:56.061246 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa72766e_f4b6_4ddd_9c8d_ad30edb83638.slice/crio-788ae3e93c4bea38769a0af4fdfe24c249bc0ff04ae53a2b2f85133a1cb061e1 WatchSource:0}: Error finding container 788ae3e93c4bea38769a0af4fdfe24c249bc0ff04ae53a2b2f85133a1cb061e1: Status 404 returned error can't find the container with id 788ae3e93c4bea38769a0af4fdfe24c249bc0ff04ae53a2b2f85133a1cb061e1 Nov 01 22:37:56.064090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:56.064065 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t"] Nov 01 22:37:56.100889 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:56.100856 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv"] Nov 01 22:37:56.105394 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:56.105371 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e4b83ffd-openshift-default-5569cb4c85-9gvzv"] Nov 01 22:37:56.912151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:56.900164 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0321f7c9-74fa-470a-87e5-88e1f9b21ca1" path="/var/lib/kubelet/pods/0321f7c9-74fa-470a-87e5-88e1f9b21ca1/volumes" Nov 01 22:37:56.912151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:56.901079 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19aa924b-c7f5-4622-9c6f-865fc6e33897" path="/var/lib/kubelet/pods/19aa924b-c7f5-4622-9c6f-865fc6e33897/volumes" Nov 01 22:37:56.912151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:56.904597 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb3ce252-0ab4-499d-abac-fed6118d1273" path="/var/lib/kubelet/pods/fb3ce252-0ab4-499d-abac-fed6118d1273/volumes" Nov 01 22:37:57.068227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.068187 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" event={"ID":"aa72766e-f4b6-4ddd-9c8d-ad30edb83638","Type":"ContainerStarted","Data":"8b3854168aca8a06c48fee831d271b7b6f02dcb6eff43467cef9ae755ddc4dc8"} Nov 01 22:37:57.068710 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.068254 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" event={"ID":"aa72766e-f4b6-4ddd-9c8d-ad30edb83638","Type":"ContainerStarted","Data":"788ae3e93c4bea38769a0af4fdfe24c249bc0ff04ae53a2b2f85133a1cb061e1"} Nov 01 22:37:57.068710 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.068420 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:37:57.093401 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.093350 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" podStartSLOduration=1.261169921 podStartE2EDuration="2.093336243s" totalImagesPullingTime="832.166322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:57.089474839 +0000 UTC m=+880.873913787" watchObservedRunningTime="2026-11-01 22:37:57.093336243 +0000 UTC m=+880.877775165" Nov 01 22:37:57.749069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.749028 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh"] Nov 01 22:37:57.753820 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.753803 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:37:57.761880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.761852 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-10a1d9\"" Nov 01 22:37:57.763309 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.763283 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh"] Nov 01 22:37:57.773584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.773559 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v"] Nov 01 22:37:57.778734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.778713 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:37:57.789999 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.789676 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v"] Nov 01 22:37:57.837609 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.837571 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-10a1d9-dockercfg-9nzw5\"" Nov 01 22:37:57.971134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.968934 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e35591-1c5a-485f-aea5-064e48802a9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh\" (UID: \"a1e35591-1c5a-485f-aea5-064e48802a9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:37:57.971134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.969136 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcwcs\" (UniqueName: \"kubernetes.io/projected/a1e35591-1c5a-485f-aea5-064e48802a9b-kube-api-access-vcwcs\") pod \"payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh\" (UID: \"a1e35591-1c5a-485f-aea5-064e48802a9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:37:57.971134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.969318 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxxgj\" (UniqueName: \"kubernetes.io/projected/42b08c73-7108-40f6-b0e0-456446484085-kube-api-access-pxxgj\") pod \"payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v\" (UID: \"42b08c73-7108-40f6-b0e0-456446484085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:37:57.971134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:57.969357 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b08c73-7108-40f6-b0e0-456446484085-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v\" (UID: \"42b08c73-7108-40f6-b0e0-456446484085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:37:58.077181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.076423 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e35591-1c5a-485f-aea5-064e48802a9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh\" (UID: \"a1e35591-1c5a-485f-aea5-064e48802a9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:37:58.077181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.076616 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwcs\" (UniqueName: \"kubernetes.io/projected/a1e35591-1c5a-485f-aea5-064e48802a9b-kube-api-access-vcwcs\") pod \"payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh\" (UID: \"a1e35591-1c5a-485f-aea5-064e48802a9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:37:58.077181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.076752 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxxgj\" (UniqueName: \"kubernetes.io/projected/42b08c73-7108-40f6-b0e0-456446484085-kube-api-access-pxxgj\") pod \"payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v\" (UID: \"42b08c73-7108-40f6-b0e0-456446484085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:37:58.077181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.076787 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b08c73-7108-40f6-b0e0-456446484085-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v\" (UID: \"42b08c73-7108-40f6-b0e0-456446484085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:37:58.077821 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.077734 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b08c73-7108-40f6-b0e0-456446484085-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v\" (UID: \"42b08c73-7108-40f6-b0e0-456446484085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:37:58.174150 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.138546 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e35591-1c5a-485f-aea5-064e48802a9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh\" (UID: \"a1e35591-1c5a-485f-aea5-064e48802a9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:37:58.174150 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.146692 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxxgj\" (UniqueName: \"kubernetes.io/projected/42b08c73-7108-40f6-b0e0-456446484085-kube-api-access-pxxgj\") pod \"payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v\" (UID: \"42b08c73-7108-40f6-b0e0-456446484085\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:37:58.174150 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.155741 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcwcs\" (UniqueName: \"kubernetes.io/projected/a1e35591-1c5a-485f-aea5-064e48802a9b-kube-api-access-vcwcs\") pod \"payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh\" (UID: \"a1e35591-1c5a-485f-aea5-064e48802a9b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:37:58.374810 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.374714 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:37:58.421735 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.416427 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:37:58.801772 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.801747 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh"] Nov 01 22:37:58.802747 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:37:58.802723 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e35591_1c5a_485f_aea5_064e48802a9b.slice/crio-b36e389c1cb4b9d97f62991311b3d120f508a8bea40e800170024045a69250f3 WatchSource:0}: Error finding container b36e389c1cb4b9d97f62991311b3d120f508a8bea40e800170024045a69250f3: Status 404 returned error can't find the container with id b36e389c1cb4b9d97f62991311b3d120f508a8bea40e800170024045a69250f3 Nov 01 22:37:58.823888 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:37:58.823864 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b08c73_7108_40f6_b0e0_456446484085.slice/crio-750ff3d3bf1f72a0abde505f620ce59f53d84ef09d6f02f933d6050fa5343c61 WatchSource:0}: Error finding container 750ff3d3bf1f72a0abde505f620ce59f53d84ef09d6f02f933d6050fa5343c61: Status 404 returned error can't find the container with id 750ff3d3bf1f72a0abde505f620ce59f53d84ef09d6f02f933d6050fa5343c61 Nov 01 22:37:58.825519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:58.825499 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v"] Nov 01 22:37:59.192182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:59.182258 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" event={"ID":"a1e35591-1c5a-485f-aea5-064e48802a9b","Type":"ContainerStarted","Data":"31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a"} Nov 01 22:37:59.192182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:59.182388 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" event={"ID":"a1e35591-1c5a-485f-aea5-064e48802a9b","Type":"ContainerStarted","Data":"b36e389c1cb4b9d97f62991311b3d120f508a8bea40e800170024045a69250f3"} Nov 01 22:37:59.192182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:59.183838 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:37:59.192182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:59.186904 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" event={"ID":"42b08c73-7108-40f6-b0e0-456446484085","Type":"ContainerStarted","Data":"b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a"} Nov 01 22:37:59.192182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:59.186983 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" event={"ID":"42b08c73-7108-40f6-b0e0-456446484085","Type":"ContainerStarted","Data":"750ff3d3bf1f72a0abde505f620ce59f53d84ef09d6f02f933d6050fa5343c61"} Nov 01 22:37:59.192182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:59.188441 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:37:59.208176 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:59.205679 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" podStartSLOduration=2.2056548559999998 podStartE2EDuration="2.205654856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:59.203595117 +0000 UTC m=+882.988034038" watchObservedRunningTime="2026-11-01 22:37:59.205654856 +0000 UTC m=+882.990093788" Nov 01 22:37:59.255131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:37:59.254381 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" podStartSLOduration=2.2543589969999998 podStartE2EDuration="2.254358997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:59.240868078 +0000 UTC m=+883.025307011" watchObservedRunningTime="2026-11-01 22:37:59.254358997 +0000 UTC m=+883.038797919" Nov 01 22:38:02.927342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:02.927303 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:38:03.180488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:03.180401 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:38:11.206836 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:11.206804 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:38:11.207253 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:11.206880 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:38:12.557401 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.557370 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc"] Nov 01 22:38:12.565081 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.565057 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw"] Nov 01 22:38:12.565253 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.565192 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:38:12.570259 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.570237 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:38:12.578338 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.578311 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-fa8cda\"" Nov 01 22:38:12.578959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.578936 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-fa8cda-dockercfg-66grd\"" Nov 01 22:38:12.586265 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.586225 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc"] Nov 01 22:38:12.600387 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.593583 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw"] Nov 01 22:38:12.704134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.702994 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f22ca4-6c61-433b-9994-46af11ec66e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc\" (UID: \"b6f22ca4-6c61-433b-9994-46af11ec66e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:38:12.704134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.703148 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr4sw\" (UniqueName: \"kubernetes.io/projected/dd9fe434-9161-4eb4-9aa2-795f3d28a633-kube-api-access-hr4sw\") pod \"payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw\" (UID: \"dd9fe434-9161-4eb4-9aa2-795f3d28a633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:38:12.704134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.703305 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7c6b\" (UniqueName: \"kubernetes.io/projected/b6f22ca4-6c61-433b-9994-46af11ec66e4-kube-api-access-h7c6b\") pod \"payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc\" (UID: \"b6f22ca4-6c61-433b-9994-46af11ec66e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:38:12.704134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.703496 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9fe434-9161-4eb4-9aa2-795f3d28a633-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw\" (UID: \"dd9fe434-9161-4eb4-9aa2-795f3d28a633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:38:12.944133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.937181 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f22ca4-6c61-433b-9994-46af11ec66e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc\" (UID: \"b6f22ca4-6c61-433b-9994-46af11ec66e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:38:12.944133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.938832 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f22ca4-6c61-433b-9994-46af11ec66e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc\" (UID: \"b6f22ca4-6c61-433b-9994-46af11ec66e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:38:12.978310 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.972443 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr4sw\" (UniqueName: \"kubernetes.io/projected/dd9fe434-9161-4eb4-9aa2-795f3d28a633-kube-api-access-hr4sw\") pod \"payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw\" (UID: \"dd9fe434-9161-4eb4-9aa2-795f3d28a633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:38:12.978310 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.972645 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7c6b\" (UniqueName: \"kubernetes.io/projected/b6f22ca4-6c61-433b-9994-46af11ec66e4-kube-api-access-h7c6b\") pod \"payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc\" (UID: \"b6f22ca4-6c61-433b-9994-46af11ec66e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:38:12.978310 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.972873 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9fe434-9161-4eb4-9aa2-795f3d28a633-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw\" (UID: \"dd9fe434-9161-4eb4-9aa2-795f3d28a633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:38:12.978310 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:12.973799 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9fe434-9161-4eb4-9aa2-795f3d28a633-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw\" (UID: \"dd9fe434-9161-4eb4-9aa2-795f3d28a633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:38:13.070186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:13.066943 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7c6b\" (UniqueName: \"kubernetes.io/projected/b6f22ca4-6c61-433b-9994-46af11ec66e4-kube-api-access-h7c6b\") pod \"payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc\" (UID: \"b6f22ca4-6c61-433b-9994-46af11ec66e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:38:13.079981 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:13.079941 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr4sw\" (UniqueName: \"kubernetes.io/projected/dd9fe434-9161-4eb4-9aa2-795f3d28a633-kube-api-access-hr4sw\") pod \"payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw\" (UID: \"dd9fe434-9161-4eb4-9aa2-795f3d28a633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:38:13.308556 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:13.308471 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:38:13.360507 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:13.360476 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:38:13.723156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:13.723095 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw"] Nov 01 22:38:13.723590 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:38:13.723397 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9fe434_9161_4eb4_9aa2_795f3d28a633.slice/crio-006a95ea802005cbfd4b3844b9d29ddccf0804aaefe9dfe0256652a556c107ff WatchSource:0}: Error finding container 006a95ea802005cbfd4b3844b9d29ddccf0804aaefe9dfe0256652a556c107ff: Status 404 returned error can't find the container with id 006a95ea802005cbfd4b3844b9d29ddccf0804aaefe9dfe0256652a556c107ff Nov 01 22:38:13.797739 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:13.797709 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc"] Nov 01 22:38:13.798278 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:38:13.798249 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f22ca4_6c61_433b_9994_46af11ec66e4.slice/crio-d2fd2bc2432f22c901691e3acbd65359ae29e42dfd177e5f4ed563be65645ede WatchSource:0}: Error finding container d2fd2bc2432f22c901691e3acbd65359ae29e42dfd177e5f4ed563be65645ede: Status 404 returned error can't find the container with id d2fd2bc2432f22c901691e3acbd65359ae29e42dfd177e5f4ed563be65645ede Nov 01 22:38:14.692408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:14.678888 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" event={"ID":"dd9fe434-9161-4eb4-9aa2-795f3d28a633","Type":"ContainerStarted","Data":"360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1"} Nov 01 22:38:14.692408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:14.678960 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" event={"ID":"dd9fe434-9161-4eb4-9aa2-795f3d28a633","Type":"ContainerStarted","Data":"006a95ea802005cbfd4b3844b9d29ddccf0804aaefe9dfe0256652a556c107ff"} Nov 01 22:38:14.735066 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:14.718852 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:38:14.822132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:14.818938 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" podStartSLOduration=2.818916145 podStartE2EDuration="2.818916145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:14.806429972 +0000 UTC m=+898.590868922" watchObservedRunningTime="2026-11-01 22:38:14.818916145 +0000 UTC m=+898.603355068" Nov 01 22:38:14.846358 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:14.844919 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" event={"ID":"b6f22ca4-6c61-433b-9994-46af11ec66e4","Type":"ContainerStarted","Data":"a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce"} Nov 01 22:38:14.846358 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:14.844988 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" event={"ID":"b6f22ca4-6c61-433b-9994-46af11ec66e4","Type":"ContainerStarted","Data":"d2fd2bc2432f22c901691e3acbd65359ae29e42dfd177e5f4ed563be65645ede"} Nov 01 22:38:14.846358 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:14.846279 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:38:19.283684 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:38:19.283643 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:38:22.520134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:22.517126 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" podStartSLOduration=10.517091041 podStartE2EDuration="10.517091041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:14.875998983 +0000 UTC m=+898.660437906" watchObservedRunningTime="2026-11-01 22:38:22.517091041 +0000 UTC m=+906.301529964" Nov 01 22:38:22.702645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:22.702579 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f"] Nov 01 22:38:23.010344 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:23.010303 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f"] Nov 01 22:38:23.010853 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:23.010640 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:38:23.024080 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:23.018155 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-41b8f5\"" Nov 01 22:38:23.024080 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:23.022129 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-41b8f5-dockercfg-m74kf\"" Nov 01 22:38:23.126134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:23.124840 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71276961-b300-4137-8f01-eb480fa73659-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f\" (UID: \"71276961-b300-4137-8f01-eb480fa73659\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:38:23.126134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:23.125008 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2zgf\" (UniqueName: \"kubernetes.io/projected/71276961-b300-4137-8f01-eb480fa73659-kube-api-access-c2zgf\") pod \"payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f\" (UID: \"71276961-b300-4137-8f01-eb480fa73659\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:38:23.227584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:23.227542 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71276961-b300-4137-8f01-eb480fa73659-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f\" (UID: \"71276961-b300-4137-8f01-eb480fa73659\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:38:23.240130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:23.227937 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2zgf\" (UniqueName: \"kubernetes.io/projected/71276961-b300-4137-8f01-eb480fa73659-kube-api-access-c2zgf\") pod \"payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f\" (UID: \"71276961-b300-4137-8f01-eb480fa73659\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:38:23.280188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:23.262932 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71276961-b300-4137-8f01-eb480fa73659-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f\" (UID: \"71276961-b300-4137-8f01-eb480fa73659\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:38:23.311542 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:23.305914 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2zgf\" (UniqueName: \"kubernetes.io/projected/71276961-b300-4137-8f01-eb480fa73659-kube-api-access-c2zgf\") pod \"payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f\" (UID: \"71276961-b300-4137-8f01-eb480fa73659\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:38:23.347663 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:23.346376 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:38:24.772146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:24.766256 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f"] Nov 01 22:38:25.483135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:25.478195 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" event={"ID":"71276961-b300-4137-8f01-eb480fa73659","Type":"ContainerStarted","Data":"307f758967e7ffdb9ef1abdc9be1be04e7bb2af3d1df81ea6bffda8958222d63"} Nov 01 22:38:25.483135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:25.478267 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" event={"ID":"71276961-b300-4137-8f01-eb480fa73659","Type":"ContainerStarted","Data":"920178cee1784b3a84aa856040aabc1046f2800b93c5de542c41e81cbfcf5135"} Nov 01 22:38:25.483135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:25.479524 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:38:25.612134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:25.611295 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" podStartSLOduration=3.611272365 podStartE2EDuration="3.611272365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:25.525836022 +0000 UTC m=+909.310274945" watchObservedRunningTime="2026-11-01 22:38:25.611272365 +0000 UTC m=+909.395711287" Nov 01 22:38:25.904132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:25.893413 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:38:26.881131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:26.877463 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:38:34.860366 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:34.860317 6466 patch_prober.go:29] interesting pod/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:34.860922 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:34.860415 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:37.653098 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:37.653018 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:38:39.187026 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:39.185195 6466 generic.go:356] "Generic (PLEG): container finished" podID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerID="5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366" exitCode=0 Nov 01 22:38:39.187026 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:39.185499 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" event={"ID":"7d8ca67a-a7ed-4a7c-905e-f02763779799","Type":"ContainerDied","Data":"5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366"} Nov 01 22:38:39.187026 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:39.186028 6466 scope.go:122] "RemoveContainer" containerID="5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366" Nov 01 22:38:39.189458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:39.189418 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:38:39.189748 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:39.189719 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:38:39.189968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:39.189930 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:38:40.042318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.042239 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:38:40.042318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.042290 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 01 22:38:40.196234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.196200 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" event={"ID":"7d8ca67a-a7ed-4a7c-905e-f02763779799","Type":"ContainerStarted","Data":"85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba"} Nov 01 22:38:40.197918 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.197892 6466 generic.go:356] "Generic (PLEG): container finished" podID="80391b86-89a6-41db-8baa-59c62315596b" containerID="4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8" exitCode=0 Nov 01 22:38:40.198036 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.197963 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" event={"ID":"80391b86-89a6-41db-8baa-59c62315596b","Type":"ContainerDied","Data":"4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8"} Nov 01 22:38:40.198419 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.198400 6466 scope.go:122] "RemoveContainer" containerID="4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8" Nov 01 22:38:40.199767 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.199750 6466 generic.go:356] "Generic (PLEG): container finished" podID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerID="a0e350bd90b75c131e38853e602f0a1f838db8a93983274d3106b3544cf9c217" exitCode=0 Nov 01 22:38:40.199836 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.199779 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" event={"ID":"1f1280b1-f9ea-489c-8b06-f595fae26903","Type":"ContainerDied","Data":"a0e350bd90b75c131e38853e602f0a1f838db8a93983274d3106b3544cf9c217"} Nov 01 22:38:40.200038 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.200015 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:38:40.200077 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.200065 6466 scope.go:122] "RemoveContainer" containerID="a0e350bd90b75c131e38853e602f0a1f838db8a93983274d3106b3544cf9c217" Nov 01 22:38:40.200125 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.200074 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:38:40.200164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.200122 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:38:40.201419 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.201392 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:38:40.201513 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.201461 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:38:40.201513 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:40.201495 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:38:41.158131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:41.155384 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:38:41.158131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:41.157519 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 22:38:41.158131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:41.157576 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 22:38:41.309088 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:41.293649 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" event={"ID":"80391b86-89a6-41db-8baa-59c62315596b","Type":"ContainerStarted","Data":"1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6"} Nov 01 22:38:41.358134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:41.358000 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" event={"ID":"1f1280b1-f9ea-489c-8b06-f595fae26903","Type":"ContainerStarted","Data":"739cbf8f725203790b4529424cebe4cfda23a0a58c723ae12ec61a816f04724b"} Nov 01 22:38:41.407617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:41.406547 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:38:41.407617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:41.406716 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:38:41.407617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:41.406775 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 01 22:38:42.121571 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:42.121529 6466 patch_prober.go:29] interesting pod/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 22:38:42.121798 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:42.121582 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 22:38:42.121879 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:42.121806 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:38:42.176132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:42.164384 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 22:38:42.176132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:42.164452 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 22:38:42.428667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:42.428631 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:38:42.429342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:42.429310 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 01 22:38:43.119596 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:43.119553 6466 patch_prober.go:29] interesting pod/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 22:38:43.119946 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:43.119907 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 22:38:43.155137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:43.150604 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 22:38:43.155137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:43.150818 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 22:38:43.427155 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:43.424502 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:38:43.427155 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:43.424583 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 01 22:38:44.085132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.071361 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:44.118630 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.118074 6466 patch_prober.go:29] interesting pod/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 22:38:44.118630 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.118244 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 22:38:44.187722 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.187691 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 22:38:44.188013 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.187985 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 22:38:44.240129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.220164 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9"] Nov 01 22:38:44.357373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.334466 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:44.476133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.427207 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:44.476133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.445722 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:44.476519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.476471 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:38:44.476606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.476539 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 01 22:38:44.811489 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.811449 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:38:44.967435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.967399 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lhnc\" (UniqueName: \"kubernetes.io/projected/ee812bbc-878f-4308-a1dc-e277443bcbcf-kube-api-access-6lhnc\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9\" (UID: \"ee812bbc-878f-4308-a1dc-e277443bcbcf\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:38:44.995838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:44.995805 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee812bbc-878f-4308-a1dc-e277443bcbcf-plugins-config-volume\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9\" (UID: \"ee812bbc-878f-4308-a1dc-e277443bcbcf\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:38:45.067220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.041474 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-425870c1\"" Nov 01 22:38:45.163144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.153611 6466 patch_prober.go:29] interesting pod/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 22:38:45.163144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.153836 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 22:38:45.167520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.167490 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 22:38:45.167741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.167714 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 22:38:45.209552 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.209512 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:38:45.212654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.212624 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9"] Nov 01 22:38:45.240729 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.240699 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lhnc\" (UniqueName: \"kubernetes.io/projected/ee812bbc-878f-4308-a1dc-e277443bcbcf-kube-api-access-6lhnc\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9\" (UID: \"ee812bbc-878f-4308-a1dc-e277443bcbcf\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:38:45.241824 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.241793 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee812bbc-878f-4308-a1dc-e277443bcbcf-plugins-config-volume\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9\" (UID: \"ee812bbc-878f-4308-a1dc-e277443bcbcf\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:38:45.284346 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.247781 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee812bbc-878f-4308-a1dc-e277443bcbcf-plugins-config-volume\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9\" (UID: \"ee812bbc-878f-4308-a1dc-e277443bcbcf\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:38:45.352200 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.332762 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lhnc\" (UniqueName: \"kubernetes.io/projected/ee812bbc-878f-4308-a1dc-e277443bcbcf-kube-api-access-6lhnc\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9\" (UID: \"ee812bbc-878f-4308-a1dc-e277443bcbcf\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:38:45.352200 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.343390 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:45.375134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.352818 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:38:45.434628 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.434585 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:45.435376 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.435352 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:38:45.456333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.456300 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:38:45.456609 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.456583 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 01 22:38:45.598998 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.598967 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:38:45.926659 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.872235 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:38:45.974133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:45.965873 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:46.129946 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:46.129492 6466 patch_prober.go:29] interesting pod/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 22:38:46.129946 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:46.129552 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 22:38:46.167833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:46.156473 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 22:38:46.167833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:46.156525 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 22:38:46.283201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:46.230813 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:46.428377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:46.428344 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:38:46.428657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:46.428629 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 01 22:38:46.610671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:46.610550 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:46.642640 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:46.642596 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:38:46.933267 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:46.933225 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:46.943447 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:46.939786 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:38:47.193263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:47.130614 6466 patch_prober.go:29] interesting pod/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 22:38:47.515226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:47.426718 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 22:38:47.661294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:47.645465 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 22:38:47.661294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:47.645531 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 22:38:47.661294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:47.660202 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:38:47.727136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:47.697248 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:38:47.771198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:47.730623 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:38:47.771198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:47.730680 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 01 22:38:47.932156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:47.834700 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-d53514"] Nov 01 22:38:48.030828 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.027529 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:38:48.030828 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.027889 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.320181 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:38:48.319921 6466 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee812bbc_878f_4308_a1dc_e277443bcbcf.slice/crio-6b636773dfe77baf7fb48f55c02ee1cf4ca06537c8688bd9d623fef8f502cf73\": RecentStats: unable to find data in memory cache]" Nov 01 22:38:48.412999 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.412956 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:38:48.413214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.413020 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 01 22:38:48.416446 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.416379 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.435168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.425805 6466 patch_prober.go:29] interesting pod/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 22:38:48.435433 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.435406 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 22:38:48.435542 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.431805 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-d53514" Nov 01 22:38:48.449313 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.434526 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 22:38:48.449313 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.438156 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 22:38:48.449313 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.431085 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:38:48.449313 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.438417 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-d53514"] Nov 01 22:38:48.449313 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.438473 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:38:48.520044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.515813 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vrfk9\"" Nov 01 22:38:48.605043 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.566482 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:38:48.607193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.606621 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9"] Nov 01 22:38:48.629026 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.628982 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lp5p\" (UniqueName: \"kubernetes.io/projected/8d278cc3-6f19-43f6-8741-06cb269d551d-kube-api-access-6lp5p\") pod \"test-curl-7024-d53514\" (UID: \"8d278cc3-6f19-43f6-8741-06cb269d551d\") " pod="opendatahub/test-curl-7024-d53514" Nov 01 22:38:48.731492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.731459 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lp5p\" (UniqueName: \"kubernetes.io/projected/8d278cc3-6f19-43f6-8741-06cb269d551d-kube-api-access-6lp5p\") pod \"test-curl-7024-d53514\" (UID: \"8d278cc3-6f19-43f6-8741-06cb269d551d\") " pod="opendatahub/test-curl-7024-d53514" Nov 01 22:38:48.770145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.755939 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lp5p\" (UniqueName: \"kubernetes.io/projected/8d278cc3-6f19-43f6-8741-06cb269d551d-kube-api-access-6lp5p\") pod \"test-curl-7024-d53514\" (UID: \"8d278cc3-6f19-43f6-8741-06cb269d551d\") " pod="opendatahub/test-curl-7024-d53514" Nov 01 22:38:48.823754 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.822410 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:38:48.899135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:48.898416 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-d53514" Nov 01 22:38:49.121693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:49.121249 6466 patch_prober.go:29] interesting pod/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 22:38:49.121693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:49.121312 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 22:38:49.201526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:49.198864 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" event={"ID":"ee812bbc-878f-4308-a1dc-e277443bcbcf","Type":"ContainerStarted","Data":"5a90e98cb3cd17ec81ac4de3a8c3bff6bdc23aa9f95854a0fd321f3fde71b7d6"} Nov 01 22:38:49.201526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:49.198929 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" event={"ID":"ee812bbc-878f-4308-a1dc-e277443bcbcf","Type":"ContainerStarted","Data":"6b636773dfe77baf7fb48f55c02ee1cf4ca06537c8688bd9d623fef8f502cf73"} Nov 01 22:38:49.201526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:49.199566 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:38:49.219040 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:49.219005 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:38:49.222224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:49.222160 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" podStartSLOduration=5.222140481 podStartE2EDuration="5.222140481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:49.215860558 +0000 UTC m=+933.000299480" watchObservedRunningTime="2026-11-01 22:38:49.222140481 +0000 UTC m=+933.006579404" Nov 01 22:38:49.224846 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:49.224821 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:38:49.409513 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:49.409476 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:38:49.410000 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:49.409537 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 01 22:38:49.430269 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:38:49.429234 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:38:50.150012 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:50.142576 6466 patch_prober.go:29] interesting pod/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 22:38:50.150012 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:50.142635 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 22:38:50.157550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:50.157506 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-d53514"] Nov 01 22:38:50.244542 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:50.244495 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-d53514" event={"ID":"8d278cc3-6f19-43f6-8741-06cb269d551d","Type":"ContainerStarted","Data":"a1f446e9e89d2411e32061e8bb6ebec43ed3be4d42f8a4ba6ad125ee6e8aebb9"} Nov 01 22:38:50.422165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:50.407884 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:38:50.422165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:50.407943 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 01 22:38:51.352708 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:51.350314 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:38:51.363455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:51.363425 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:38:51.416555 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:51.416520 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-d53514" event={"ID":"8d278cc3-6f19-43f6-8741-06cb269d551d","Type":"ContainerStarted","Data":"86a826e4102339f58c366a7afdb91bb601c36a2679a98af54dc0c8e765d06d0b"} Nov 01 22:38:51.450150 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:51.446173 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-d53514" podStartSLOduration=4.446150438 podStartE2EDuration="4.446150438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:51.43699132 +0000 UTC m=+935.221430242" watchObservedRunningTime="2026-11-01 22:38:51.446150438 +0000 UTC m=+935.230589354" Nov 01 22:38:51.475882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:51.469329 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:38:51.475882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:51.471701 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:38:51.476358 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:51.476334 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:38:52.197336 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.197301 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-d53514"] Nov 01 22:38:52.206805 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.206771 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-d53514"] Nov 01 22:38:52.353392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.353363 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-399fa7"] Nov 01 22:38:52.354140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.354125 6466 cpu_manager.go:414] "containerMap: removing container" podUID="8d278cc3-6f19-43f6-8741-06cb269d551d" containerName="test-curl-7024-d53514" Nov 01 22:38:52.354187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.354141 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d278cc3-6f19-43f6-8741-06cb269d551d" containerName="test-curl-7024-d53514" Nov 01 22:38:52.354320 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.354311 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d278cc3-6f19-43f6-8741-06cb269d551d" containerName="test-curl-7024-d53514" Nov 01 22:38:52.357118 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.357086 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-399fa7" Nov 01 22:38:52.366985 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.366965 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-399fa7"] Nov 01 22:38:52.427728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.427694 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvss\" (UniqueName: \"kubernetes.io/projected/7bd5d80c-8550-41ef-9223-7547e2d4d4fd-kube-api-access-6lvss\") pod \"test-curl-7024-399fa7\" (UID: \"7bd5d80c-8550-41ef-9223-7547e2d4d4fd\") " pod="opendatahub/test-curl-7024-399fa7" Nov 01 22:38:52.529163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.529070 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvss\" (UniqueName: \"kubernetes.io/projected/7bd5d80c-8550-41ef-9223-7547e2d4d4fd-kube-api-access-6lvss\") pod \"test-curl-7024-399fa7\" (UID: \"7bd5d80c-8550-41ef-9223-7547e2d4d4fd\") " pod="opendatahub/test-curl-7024-399fa7" Nov 01 22:38:52.541364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.541334 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvss\" (UniqueName: \"kubernetes.io/projected/7bd5d80c-8550-41ef-9223-7547e2d4d4fd-kube-api-access-6lvss\") pod \"test-curl-7024-399fa7\" (UID: \"7bd5d80c-8550-41ef-9223-7547e2d4d4fd\") " pod="opendatahub/test-curl-7024-399fa7" Nov 01 22:38:52.672098 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.672062 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-399fa7" Nov 01 22:38:52.957934 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:38:52.957900 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd5d80c_8550_41ef_9223_7547e2d4d4fd.slice/crio-6eaef138e5b4e31c8b56aa206f3a944f835635f7b8fa6b661d2c1a1514ca8085 WatchSource:0}: Error finding container 6eaef138e5b4e31c8b56aa206f3a944f835635f7b8fa6b661d2c1a1514ca8085: Status 404 returned error can't find the container with id 6eaef138e5b4e31c8b56aa206f3a944f835635f7b8fa6b661d2c1a1514ca8085 Nov 01 22:38:52.958246 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:52.958225 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-399fa7"] Nov 01 22:38:53.436294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:53.436258 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-399fa7" event={"ID":"7bd5d80c-8550-41ef-9223-7547e2d4d4fd","Type":"ContainerStarted","Data":"417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249"} Nov 01 22:38:53.436456 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:53.436311 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-399fa7" event={"ID":"7bd5d80c-8550-41ef-9223-7547e2d4d4fd","Type":"ContainerStarted","Data":"6eaef138e5b4e31c8b56aa206f3a944f835635f7b8fa6b661d2c1a1514ca8085"} Nov 01 22:38:53.436456 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:53.436360 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-d53514" podUID="8d278cc3-6f19-43f6-8741-06cb269d551d" containerName="test-curl-7024-d53514" containerID="cri-o://86a826e4102339f58c366a7afdb91bb601c36a2679a98af54dc0c8e765d06d0b" gracePeriod=2 Nov 01 22:38:53.455024 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:53.454977 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-399fa7" podStartSLOduration=1.454961583 podStartE2EDuration="1.454961583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:53.450927309 +0000 UTC m=+937.235366231" watchObservedRunningTime="2026-11-01 22:38:53.454961583 +0000 UTC m=+937.239400503" Nov 01 22:38:54.224270 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.224235 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-399fa7"] Nov 01 22:38:54.228050 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.228018 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-399fa7"] Nov 01 22:38:54.312388 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.312354 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42"] Nov 01 22:38:54.313872 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.313844 6466 cpu_manager.go:414] "containerMap: removing container" podUID="7bd5d80c-8550-41ef-9223-7547e2d4d4fd" containerName="test-curl-7024-399fa7" Nov 01 22:38:54.313872 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.313870 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd5d80c-8550-41ef-9223-7547e2d4d4fd" containerName="test-curl-7024-399fa7" Nov 01 22:38:54.314091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.314062 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd5d80c-8550-41ef-9223-7547e2d4d4fd" containerName="test-curl-7024-399fa7" Nov 01 22:38:54.318673 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.317625 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.324841 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.323340 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-425870c1-serving-cert\"" Nov 01 22:38:54.325011 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.324852 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42"] Nov 01 22:38:54.379143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.379076 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-6d6e39"] Nov 01 22:38:54.391991 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.391952 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-6d6e39"] Nov 01 22:38:54.392217 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.392049 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-6d6e39" Nov 01 22:38:54.400773 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.400734 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe38402-a86d-4392-bea5-0452d9bebcae-openshift-service-ca\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.400959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.400836 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhmtf\" (UniqueName: \"kubernetes.io/projected/5fe38402-a86d-4392-bea5-0452d9bebcae-kube-api-access-xhmtf\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.402434 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.402402 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-metrics-certs\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.402556 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.402505 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-maas-api-tls\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.402556 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.402548 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhp4\" (UniqueName: \"kubernetes.io/projected/32f2cf3d-19ea-44e5-ad2e-6919bfe411c0-kube-api-access-qxhp4\") pod \"test-curl-7024-6d6e39\" (UID: \"32f2cf3d-19ea-44e5-ad2e-6919bfe411c0\") " pod="opendatahub/test-curl-7024-6d6e39" Nov 01 22:38:54.510161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.507566 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-maas-api-tls\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.510161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.507700 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhp4\" (UniqueName: \"kubernetes.io/projected/32f2cf3d-19ea-44e5-ad2e-6919bfe411c0-kube-api-access-qxhp4\") pod \"test-curl-7024-6d6e39\" (UID: \"32f2cf3d-19ea-44e5-ad2e-6919bfe411c0\") " pod="opendatahub/test-curl-7024-6d6e39" Nov 01 22:38:54.510161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.508031 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe38402-a86d-4392-bea5-0452d9bebcae-openshift-service-ca\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.510161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.508125 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhmtf\" (UniqueName: \"kubernetes.io/projected/5fe38402-a86d-4392-bea5-0452d9bebcae-kube-api-access-xhmtf\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.510161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.508324 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-metrics-certs\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.514136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.512623 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-metrics-certs\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.514136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.513267 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe38402-a86d-4392-bea5-0452d9bebcae-openshift-service-ca\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.528161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.523612 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-maas-api-tls\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.528161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.526737 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhmtf\" (UniqueName: \"kubernetes.io/projected/5fe38402-a86d-4392-bea5-0452d9bebcae-kube-api-access-xhmtf\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-47x42\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.561129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.529002 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhp4\" (UniqueName: \"kubernetes.io/projected/32f2cf3d-19ea-44e5-ad2e-6919bfe411c0-kube-api-access-qxhp4\") pod \"test-curl-7024-6d6e39\" (UID: \"32f2cf3d-19ea-44e5-ad2e-6919bfe411c0\") " pod="opendatahub/test-curl-7024-6d6e39" Nov 01 22:38:54.648733 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.648696 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:54.708298 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:54.708238 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-6d6e39" Nov 01 22:38:55.090962 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:55.088337 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42"] Nov 01 22:38:55.111040 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:38:55.110627 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe38402_a86d_4392_bea5_0452d9bebcae.slice/crio-63242528a4936f1ca7a00579c3b878e6b6892bd9fd8d77834509f6c3fd025282 WatchSource:0}: Error finding container 63242528a4936f1ca7a00579c3b878e6b6892bd9fd8d77834509f6c3fd025282: Status 404 returned error can't find the container with id 63242528a4936f1ca7a00579c3b878e6b6892bd9fd8d77834509f6c3fd025282 Nov 01 22:38:55.165796 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:55.165760 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-6d6e39"] Nov 01 22:38:55.535470 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:55.535435 6466 generic.go:356] "Generic (PLEG): container finished" podID="8d278cc3-6f19-43f6-8741-06cb269d551d" containerID="86a826e4102339f58c366a7afdb91bb601c36a2679a98af54dc0c8e765d06d0b" exitCode=137 Nov 01 22:38:55.537577 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:55.537546 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" event={"ID":"5fe38402-a86d-4392-bea5-0452d9bebcae","Type":"ContainerStarted","Data":"63242528a4936f1ca7a00579c3b878e6b6892bd9fd8d77834509f6c3fd025282"} Nov 01 22:38:55.540064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:55.540032 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-399fa7" podUID="7bd5d80c-8550-41ef-9223-7547e2d4d4fd" containerName="test-curl-7024-399fa7" containerID="cri-o://417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249" gracePeriod=2 Nov 01 22:38:55.540364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:55.540335 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-6d6e39" event={"ID":"32f2cf3d-19ea-44e5-ad2e-6919bfe411c0","Type":"ContainerStarted","Data":"d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0"} Nov 01 22:38:55.540555 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:55.540534 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-6d6e39" event={"ID":"32f2cf3d-19ea-44e5-ad2e-6919bfe411c0","Type":"ContainerStarted","Data":"7630652467c4088734f9cf0644e2b8031404a2a7ceae8b2790dcb1df272e057a"} Nov 01 22:38:55.635210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:55.635182 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-d53514" Nov 01 22:38:55.704498 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:55.704469 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d278cc3-6f19-43f6-8741-06cb269d551d-kube-api-access-6lp5p\" (UniqueName: \"kubernetes.io/projected/8d278cc3-6f19-43f6-8741-06cb269d551d-kube-api-access-6lp5p\") pod \"8d278cc3-6f19-43f6-8741-06cb269d551d\" (UID: \"8d278cc3-6f19-43f6-8741-06cb269d551d\") " Nov 01 22:38:55.706498 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:55.706471 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d278cc3-6f19-43f6-8741-06cb269d551d-kube-api-access-6lp5p" pod "8d278cc3-6f19-43f6-8741-06cb269d551d" (UID: "8d278cc3-6f19-43f6-8741-06cb269d551d"). InnerVolumeSpecName "kube-api-access-6lp5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:38:55.807756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:55.807722 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lp5p\" (UniqueName: \"kubernetes.io/projected/8d278cc3-6f19-43f6-8741-06cb269d551d-kube-api-access-6lp5p\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:56.309586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.309537 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-6d6e39" podStartSLOduration=2.309523857 podStartE2EDuration="2.309523857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:55.563282317 +0000 UTC m=+939.347721238" watchObservedRunningTime="2026-11-01 22:38:56.309523857 +0000 UTC m=+940.093962777" Nov 01 22:38:56.313534 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.313500 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-6d6e39"] Nov 01 22:38:56.319941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.319915 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-6d6e39"] Nov 01 22:38:56.349832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.349796 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d278cc3-6f19-43f6-8741-06cb269d551d" path="/var/lib/kubelet/pods/8d278cc3-6f19-43f6-8741-06cb269d551d/volumes" Nov 01 22:38:56.471646 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.471614 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-8e13db"] Nov 01 22:38:56.472659 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.472641 6466 cpu_manager.go:414] "containerMap: removing container" podUID="32f2cf3d-19ea-44e5-ad2e-6919bfe411c0" containerName="test-curl-7024-6d6e39" Nov 01 22:38:56.472727 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.472661 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f2cf3d-19ea-44e5-ad2e-6919bfe411c0" containerName="test-curl-7024-6d6e39" Nov 01 22:38:56.472787 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.472777 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f2cf3d-19ea-44e5-ad2e-6919bfe411c0" containerName="test-curl-7024-6d6e39" Nov 01 22:38:56.476878 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.476843 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-8e13db" Nov 01 22:38:56.482544 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.482522 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-8e13db"] Nov 01 22:38:56.550183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.550158 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-d53514" Nov 01 22:38:56.550706 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.550180 6466 scope.go:122] "RemoveContainer" containerID="86a826e4102339f58c366a7afdb91bb601c36a2679a98af54dc0c8e765d06d0b" Nov 01 22:38:56.552528 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.552496 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" event={"ID":"5fe38402-a86d-4392-bea5-0452d9bebcae","Type":"ContainerStarted","Data":"8b3aaa8e6215fa751ca90208160e35d391091e095d47991c149347123f3b8216"} Nov 01 22:38:56.552626 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.552579 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:38:56.574860 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.574813 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" podStartSLOduration=2.091102285 podStartE2EDuration="2.574797481s" totalImagesPullingTime="483.695196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:56.574425759 +0000 UTC m=+940.358864681" watchObservedRunningTime="2026-11-01 22:38:56.574797481 +0000 UTC m=+940.359236403" Nov 01 22:38:56.623144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.623075 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtwcg\" (UniqueName: \"kubernetes.io/projected/91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e-kube-api-access-rtwcg\") pod \"test-curl-7024-8e13db\" (UID: \"91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e\") " pod="opendatahub/test-curl-7024-8e13db" Nov 01 22:38:56.734135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.729058 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtwcg\" (UniqueName: \"kubernetes.io/projected/91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e-kube-api-access-rtwcg\") pod \"test-curl-7024-8e13db\" (UID: \"91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e\") " pod="opendatahub/test-curl-7024-8e13db" Nov 01 22:38:56.757161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.757128 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtwcg\" (UniqueName: \"kubernetes.io/projected/91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e-kube-api-access-rtwcg\") pod \"test-curl-7024-8e13db\" (UID: \"91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e\") " pod="opendatahub/test-curl-7024-8e13db" Nov 01 22:38:56.791970 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:56.791936 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-8e13db" Nov 01 22:38:57.130738 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:38:57.130707 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91b7cd1c_14f6_4c22_8b0f_ae3bbe36f09e.slice/crio-c0fecd9380e31005025b27ac715606459338dbec7bfea912eb838a63bf60842d WatchSource:0}: Error finding container c0fecd9380e31005025b27ac715606459338dbec7bfea912eb838a63bf60842d: Status 404 returned error can't find the container with id c0fecd9380e31005025b27ac715606459338dbec7bfea912eb838a63bf60842d Nov 01 22:38:57.131942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:57.131916 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-8e13db"] Nov 01 22:38:57.597780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:57.593189 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-8e13db" event={"ID":"91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e","Type":"ContainerStarted","Data":"8e82cc1fb72daa509780ce4a4ca8a795651cb5dcae1a9f52e1ae305f48b4a2ed"} Nov 01 22:38:57.597780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:57.593259 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-8e13db" event={"ID":"91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e","Type":"ContainerStarted","Data":"c0fecd9380e31005025b27ac715606459338dbec7bfea912eb838a63bf60842d"} Nov 01 22:38:57.597780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:57.593449 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-6d6e39" podUID="32f2cf3d-19ea-44e5-ad2e-6919bfe411c0" containerName="test-curl-7024-6d6e39" containerID="cri-o://d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0" gracePeriod=2 Nov 01 22:38:57.615121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:57.615058 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-8e13db" podStartSLOduration=1.615039026 podStartE2EDuration="1.615039026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:57.609263405 +0000 UTC m=+941.393702329" watchObservedRunningTime="2026-11-01 22:38:57.615039026 +0000 UTC m=+941.399477949" Nov 01 22:38:57.832894 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:57.832865 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-399fa7" Nov 01 22:38:57.956514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:57.956476 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd5d80c-8550-41ef-9223-7547e2d4d4fd-kube-api-access-6lvss\" (UniqueName: \"kubernetes.io/projected/7bd5d80c-8550-41ef-9223-7547e2d4d4fd-kube-api-access-6lvss\") pod \"7bd5d80c-8550-41ef-9223-7547e2d4d4fd\" (UID: \"7bd5d80c-8550-41ef-9223-7547e2d4d4fd\") " Nov 01 22:38:57.958594 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:57.958567 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd5d80c-8550-41ef-9223-7547e2d4d4fd-kube-api-access-6lvss" pod "7bd5d80c-8550-41ef-9223-7547e2d4d4fd" (UID: "7bd5d80c-8550-41ef-9223-7547e2d4d4fd"). InnerVolumeSpecName "kube-api-access-6lvss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:38:58.059454 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:58.059376 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lvss\" (UniqueName: \"kubernetes.io/projected/7bd5d80c-8550-41ef-9223-7547e2d4d4fd-kube-api-access-6lvss\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:58.326838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:58.326741 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-8e13db"] Nov 01 22:38:58.331825 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:58.331795 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-8e13db"] Nov 01 22:38:58.382072 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:58.382035 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd5d80c-8550-41ef-9223-7547e2d4d4fd" path="/var/lib/kubelet/pods/7bd5d80c-8550-41ef-9223-7547e2d4d4fd/volumes" Nov 01 22:38:58.601276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:58.601177 6466 generic.go:356] "Generic (PLEG): container finished" podID="7bd5d80c-8550-41ef-9223-7547e2d4d4fd" containerID="417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249" exitCode=137 Nov 01 22:38:58.601276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:58.601236 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-399fa7" Nov 01 22:38:58.601742 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:58.601279 6466 scope.go:122] "RemoveContainer" containerID="417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249" Nov 01 22:38:58.611917 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:58.611898 6466 scope.go:122] "RemoveContainer" containerID="417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249" Nov 01 22:38:58.612269 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:38:58.612244 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249\": container with ID starting with 417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249 not found: ID does not exist" containerID="417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249" Nov 01 22:38:58.612363 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:58.612278 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249"} err="failed to get container status \"417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249\": rpc error: code = NotFound desc = could not find container \"417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249\": container with ID starting with 417ed7366fb349b379f0bdb8def02f1bf37f98a21c20da779d377b5178d8e249 not found: ID does not exist" Nov 01 22:38:59.612806 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:59.612766 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-8e13db" podUID="91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e" containerName="test-curl-7024-8e13db" containerID="cri-o://8e82cc1fb72daa509780ce4a4ca8a795651cb5dcae1a9f52e1ae305f48b4a2ed" gracePeriod=2 Nov 01 22:38:59.925388 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:38:59.925358 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-6d6e39" Nov 01 22:39:00.098462 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:00.098432 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f2cf3d-19ea-44e5-ad2e-6919bfe411c0-kube-api-access-qxhp4\" (UniqueName: \"kubernetes.io/projected/32f2cf3d-19ea-44e5-ad2e-6919bfe411c0-kube-api-access-qxhp4\") pod \"32f2cf3d-19ea-44e5-ad2e-6919bfe411c0\" (UID: \"32f2cf3d-19ea-44e5-ad2e-6919bfe411c0\") " Nov 01 22:39:00.100733 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:00.100671 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f2cf3d-19ea-44e5-ad2e-6919bfe411c0-kube-api-access-qxhp4" pod "32f2cf3d-19ea-44e5-ad2e-6919bfe411c0" (UID: "32f2cf3d-19ea-44e5-ad2e-6919bfe411c0"). InnerVolumeSpecName "kube-api-access-qxhp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:00.201130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:00.201033 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxhp4\" (UniqueName: \"kubernetes.io/projected/32f2cf3d-19ea-44e5-ad2e-6919bfe411c0-kube-api-access-qxhp4\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:00.249891 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:00.249863 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:39:00.345897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:00.345862 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f2cf3d-19ea-44e5-ad2e-6919bfe411c0" path="/var/lib/kubelet/pods/32f2cf3d-19ea-44e5-ad2e-6919bfe411c0/volumes" Nov 01 22:39:00.670775 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:00.670722 6466 generic.go:356] "Generic (PLEG): container finished" podID="32f2cf3d-19ea-44e5-ad2e-6919bfe411c0" containerID="d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0" exitCode=137 Nov 01 22:39:00.671517 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:00.671370 6466 scope.go:122] "RemoveContainer" containerID="d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0" Nov 01 22:39:00.671715 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:00.671433 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-6d6e39" Nov 01 22:39:00.730749 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:00.730490 6466 scope.go:122] "RemoveContainer" containerID="d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0" Nov 01 22:39:00.730970 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:39:00.730934 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0\": container with ID starting with d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0 not found: ID does not exist" containerID="d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0" Nov 01 22:39:00.731134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:00.730970 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0"} err="failed to get container status \"d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0\": rpc error: code = NotFound desc = could not find container \"d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0\": container with ID starting with d99bdf6e99f7d7dd050d75b967717d7984d3d5c099822436a047535d324e47a0 not found: ID does not exist" Nov 01 22:39:01.682755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:01.682724 6466 generic.go:356] "Generic (PLEG): container finished" podID="91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e" containerID="8e82cc1fb72daa509780ce4a4ca8a795651cb5dcae1a9f52e1ae305f48b4a2ed" exitCode=137 Nov 01 22:39:01.851635 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:01.851609 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-8e13db" Nov 01 22:39:01.877343 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:01.877278 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e-kube-api-access-rtwcg\" (UniqueName: \"kubernetes.io/projected/91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e-kube-api-access-rtwcg\") pod \"91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e\" (UID: \"91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e\") " Nov 01 22:39:01.879372 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:01.879340 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e-kube-api-access-rtwcg" pod "91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e" (UID: "91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e"). InnerVolumeSpecName "kube-api-access-rtwcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:01.979723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:01.979689 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtwcg\" (UniqueName: \"kubernetes.io/projected/91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e-kube-api-access-rtwcg\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:02.343406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:02.343376 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e" path="/var/lib/kubelet/pods/91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e/volumes" Nov 01 22:39:02.598028 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:02.597953 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:39:02.694689 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:02.694664 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-8e13db" Nov 01 22:39:02.695913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:02.694674 6466 scope.go:122] "RemoveContainer" containerID="8e82cc1fb72daa509780ce4a4ca8a795651cb5dcae1a9f52e1ae305f48b4a2ed" Nov 01 22:39:17.587134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:17.586559 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.289409 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:39:19.289370 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:39:37.174622 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:37.174588 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4"] Nov 01 22:39:37.188247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:37.188211 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7"] Nov 01 22:39:41.952697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:41.952647 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9"] Nov 01 22:39:41.953513 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:41.953471 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" podUID="d757b14b-037a-429b-a537-268485243679" containerName="maas-api" containerID="cri-o://5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2" gracePeriod=30 Nov 01 22:39:42.189934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.176482 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" podUID="54f64bb1-139c-453e-9c86-d7289bbcbf44" containerName="payload-processing" containerID="cri-o://1000c76694aca1f4f7167cfa670b9fe21c064391822c47fe68c4f184df72a6cd" gracePeriod=25 Nov 01 22:39:42.221745 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.190371 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" podUID="3662ca88-be02-42ef-aadd-bce5e426d8bb" containerName="payload-pre-processing" containerID="cri-o://f199c08ac85ca4c7bd3bfda753113a53f9207ca75c84277bd4ea2289ae11cc9c" gracePeriod=25 Nov 01 22:39:42.694159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.691407 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:39:42.743175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.739276 6466 generic.go:356] "Generic (PLEG): container finished" podID="d757b14b-037a-429b-a537-268485243679" containerID="5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2" exitCode=0 Nov 01 22:39:42.743175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.739354 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" Nov 01 22:39:42.743175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.739471 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" event={"ID":"d757b14b-037a-429b-a537-268485243679","Type":"ContainerDied","Data":"5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2"} Nov 01 22:39:42.743175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.739520 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9" event={"ID":"d757b14b-037a-429b-a537-268485243679","Type":"ContainerDied","Data":"87f9dda7a33c54f38180febd6ab75e38553285fa4e4c8de48eaf071b40d302cf"} Nov 01 22:39:42.743175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.739547 6466 scope.go:122] "RemoveContainer" containerID="5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2" Nov 01 22:39:42.748962 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.744974 6466 generic.go:356] "Generic (PLEG): container finished" podID="3662ca88-be02-42ef-aadd-bce5e426d8bb" containerID="f199c08ac85ca4c7bd3bfda753113a53f9207ca75c84277bd4ea2289ae11cc9c" exitCode=0 Nov 01 22:39:42.748962 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.745051 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" event={"ID":"3662ca88-be02-42ef-aadd-bce5e426d8bb","Type":"ContainerDied","Data":"f199c08ac85ca4c7bd3bfda753113a53f9207ca75c84277bd4ea2289ae11cc9c"} Nov 01 22:39:42.750245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.749944 6466 generic.go:356] "Generic (PLEG): container finished" podID="54f64bb1-139c-453e-9c86-d7289bbcbf44" containerID="1000c76694aca1f4f7167cfa670b9fe21c064391822c47fe68c4f184df72a6cd" exitCode=0 Nov 01 22:39:42.750245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.750145 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" event={"ID":"54f64bb1-139c-453e-9c86-d7289bbcbf44","Type":"ContainerDied","Data":"1000c76694aca1f4f7167cfa670b9fe21c064391822c47fe68c4f184df72a6cd"} Nov 01 22:39:42.767200 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.767173 6466 scope.go:122] "RemoveContainer" containerID="5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2" Nov 01 22:39:42.767571 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:39:42.767546 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2\": container with ID starting with 5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2 not found: ID does not exist" containerID="5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2" Nov 01 22:39:42.767658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.767579 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2"} err="failed to get container status \"5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2\": rpc error: code = NotFound desc = could not find container \"5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2\": container with ID starting with 5387bdb021dd8a11c6698e8c88a47fa00fd095aa06e276da74f8b2f01dc0d8f2 not found: ID does not exist" Nov 01 22:39:42.790648 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.789466 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d757b14b-037a-429b-a537-268485243679-kube-api-access-ztfhx\" (UniqueName: \"kubernetes.io/projected/d757b14b-037a-429b-a537-268485243679-kube-api-access-ztfhx\") pod \"d757b14b-037a-429b-a537-268485243679\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " Nov 01 22:39:42.790648 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.789610 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-maas-api-tls\") pod \"d757b14b-037a-429b-a537-268485243679\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " Nov 01 22:39:42.790648 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.789831 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-metrics-certs\") pod \"d757b14b-037a-429b-a537-268485243679\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " Nov 01 22:39:42.790648 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.789945 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d757b14b-037a-429b-a537-268485243679-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d757b14b-037a-429b-a537-268485243679-openshift-service-ca\") pod \"d757b14b-037a-429b-a537-268485243679\" (UID: \"d757b14b-037a-429b-a537-268485243679\") " Nov 01 22:39:42.790648 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.790583 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d757b14b-037a-429b-a537-268485243679-openshift-service-ca" pod "d757b14b-037a-429b-a537-268485243679" (UID: "d757b14b-037a-429b-a537-268485243679"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:42.791016 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.790839 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d757b14b-037a-429b-a537-268485243679-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:42.793409 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.793383 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d757b14b-037a-429b-a537-268485243679-kube-api-access-ztfhx" pod "d757b14b-037a-429b-a537-268485243679" (UID: "d757b14b-037a-429b-a537-268485243679"). InnerVolumeSpecName "kube-api-access-ztfhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:42.793957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.793938 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-maas-api-tls" pod "d757b14b-037a-429b-a537-268485243679" (UID: "d757b14b-037a-429b-a537-268485243679"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:42.794049 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.794034 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-metrics-certs" pod "d757b14b-037a-429b-a537-268485243679" (UID: "d757b14b-037a-429b-a537-268485243679"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:42.942406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.934727 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:42.942406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.934763 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d757b14b-037a-429b-a537-268485243679-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:42.942406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.934782 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztfhx\" (UniqueName: \"kubernetes.io/projected/d757b14b-037a-429b-a537-268485243679-kube-api-access-ztfhx\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:42.988128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:42.979301 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:39:43.081985 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.081960 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:39:43.098517 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.098488 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9"] Nov 01 22:39:43.102412 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.102384 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66033bd2-5d884c49d8-44rv9"] Nov 01 22:39:43.138781 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.138744 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f64bb1-139c-453e-9c86-d7289bbcbf44-kube-api-access-4d2qb\" (UniqueName: \"kubernetes.io/projected/54f64bb1-139c-453e-9c86-d7289bbcbf44-kube-api-access-4d2qb\") pod \"54f64bb1-139c-453e-9c86-d7289bbcbf44\" (UID: \"54f64bb1-139c-453e-9c86-d7289bbcbf44\") " Nov 01 22:39:43.139220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.139193 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f64bb1-139c-453e-9c86-d7289bbcbf44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f64bb1-139c-453e-9c86-d7289bbcbf44-plugins-config-volume\") pod \"54f64bb1-139c-453e-9c86-d7289bbcbf44\" (UID: \"54f64bb1-139c-453e-9c86-d7289bbcbf44\") " Nov 01 22:39:43.139630 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.139605 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f64bb1-139c-453e-9c86-d7289bbcbf44-plugins-config-volume" pod "54f64bb1-139c-453e-9c86-d7289bbcbf44" (UID: "54f64bb1-139c-453e-9c86-d7289bbcbf44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:43.140002 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.139985 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f64bb1-139c-453e-9c86-d7289bbcbf44-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:43.140922 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.140901 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f64bb1-139c-453e-9c86-d7289bbcbf44-kube-api-access-4d2qb" pod "54f64bb1-139c-453e-9c86-d7289bbcbf44" (UID: "54f64bb1-139c-453e-9c86-d7289bbcbf44"). InnerVolumeSpecName "kube-api-access-4d2qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:43.240610 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.240574 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3662ca88-be02-42ef-aadd-bce5e426d8bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3662ca88-be02-42ef-aadd-bce5e426d8bb-plugins-config-volume\") pod \"3662ca88-be02-42ef-aadd-bce5e426d8bb\" (UID: \"3662ca88-be02-42ef-aadd-bce5e426d8bb\") " Nov 01 22:39:43.240815 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.240649 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3662ca88-be02-42ef-aadd-bce5e426d8bb-kube-api-access-c8pzc\" (UniqueName: \"kubernetes.io/projected/3662ca88-be02-42ef-aadd-bce5e426d8bb-kube-api-access-c8pzc\") pod \"3662ca88-be02-42ef-aadd-bce5e426d8bb\" (UID: \"3662ca88-be02-42ef-aadd-bce5e426d8bb\") " Nov 01 22:39:43.241011 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.240985 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3662ca88-be02-42ef-aadd-bce5e426d8bb-plugins-config-volume" pod "3662ca88-be02-42ef-aadd-bce5e426d8bb" (UID: "3662ca88-be02-42ef-aadd-bce5e426d8bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:43.241555 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.241529 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3662ca88-be02-42ef-aadd-bce5e426d8bb-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:43.241555 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.241552 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d2qb\" (UniqueName: \"kubernetes.io/projected/54f64bb1-139c-453e-9c86-d7289bbcbf44-kube-api-access-4d2qb\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:43.242873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.242843 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3662ca88-be02-42ef-aadd-bce5e426d8bb-kube-api-access-c8pzc" pod "3662ca88-be02-42ef-aadd-bce5e426d8bb" (UID: "3662ca88-be02-42ef-aadd-bce5e426d8bb"). InnerVolumeSpecName "kube-api-access-c8pzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:43.343796 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.343715 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8pzc\" (UniqueName: \"kubernetes.io/projected/3662ca88-be02-42ef-aadd-bce5e426d8bb-kube-api-access-c8pzc\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:43.759852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.759826 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" Nov 01 22:39:43.760042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.759823 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7" event={"ID":"3662ca88-be02-42ef-aadd-bce5e426d8bb","Type":"ContainerDied","Data":"670dd0dbbe41adeb359f24ebbbbf1f4b9789e47147a20fe00494be2d43246438"} Nov 01 22:39:43.760042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.759949 6466 scope.go:122] "RemoveContainer" containerID="f199c08ac85ca4c7bd3bfda753113a53f9207ca75c84277bd4ea2289ae11cc9c" Nov 01 22:39:43.761384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.761364 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" Nov 01 22:39:43.761511 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.761364 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4" event={"ID":"54f64bb1-139c-453e-9c86-d7289bbcbf44","Type":"ContainerDied","Data":"73dd54d9f157373027d8943c94ac24c70fb21d887b0853145dd0c78d25fedb6f"} Nov 01 22:39:43.772974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.772957 6466 scope.go:122] "RemoveContainer" containerID="1000c76694aca1f4f7167cfa670b9fe21c064391822c47fe68c4f184df72a6cd" Nov 01 22:39:43.805478 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.805449 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7"] Nov 01 22:39:43.819261 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.819229 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66033bd2-57b666d687-rvvc7"] Nov 01 22:39:43.841554 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.841523 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4"] Nov 01 22:39:43.859225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:43.859124 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66033bd2-6b647ff7f5-zd5w4"] Nov 01 22:39:44.342993 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:44.342963 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3662ca88-be02-42ef-aadd-bce5e426d8bb" path="/var/lib/kubelet/pods/3662ca88-be02-42ef-aadd-bce5e426d8bb/volumes" Nov 01 22:39:44.343509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:44.343494 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f64bb1-139c-453e-9c86-d7289bbcbf44" path="/var/lib/kubelet/pods/54f64bb1-139c-453e-9c86-d7289bbcbf44/volumes" Nov 01 22:39:44.343931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:44.343918 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d757b14b-037a-429b-a537-268485243679" path="/var/lib/kubelet/pods/d757b14b-037a-429b-a537-268485243679/volumes" Nov 01 22:39:49.429294 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:39:49.429251 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:39:50.664178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:50.646204 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98"] Nov 01 22:39:50.664178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:50.646494 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" containerID="cri-o://1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6" gracePeriod=30 Nov 01 22:39:51.417132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:51.388856 6466 patch_prober.go:29] interesting pod/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:39:51.417506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:51.417468 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:39:51.534137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:51.533044 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.643134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:51.637531 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dgmc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:51.643134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:51.637603 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.741189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:51.741030 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q5hw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:51.741189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:51.741119 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.076203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.073577 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ggzxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:52.076203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.074670 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.144600 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.144554 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.590134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.584265 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6"] Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603396 6466 cpu_manager.go:414] "containerMap: removing container" podUID="54f64bb1-139c-453e-9c86-d7289bbcbf44" containerName="payload-processing" Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603427 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f64bb1-139c-453e-9c86-d7289bbcbf44" containerName="payload-processing" Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603456 6466 cpu_manager.go:414] "containerMap: removing container" podUID="3662ca88-be02-42ef-aadd-bce5e426d8bb" containerName="payload-pre-processing" Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603464 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3662ca88-be02-42ef-aadd-bce5e426d8bb" containerName="payload-pre-processing" Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603505 6466 cpu_manager.go:414] "containerMap: removing container" podUID="91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e" containerName="test-curl-7024-8e13db" Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603512 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e" containerName="test-curl-7024-8e13db" Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603537 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d757b14b-037a-429b-a537-268485243679" containerName="maas-api" Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603544 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d757b14b-037a-429b-a537-268485243679" containerName="maas-api" Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603741 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f64bb1-139c-453e-9c86-d7289bbcbf44" containerName="payload-processing" Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603753 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="3662ca88-be02-42ef-aadd-bce5e426d8bb" containerName="payload-pre-processing" Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603770 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b7cd1c-14f6-4c22-8b0f-ae3bbe36f09e" containerName="test-curl-7024-8e13db" Nov 01 22:39:52.605135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.603781 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d757b14b-037a-429b-a537-268485243679" containerName="maas-api" Nov 01 22:39:52.699529 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.699492 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6"] Nov 01 22:39:52.699885 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.699865 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.748145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.729588 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-93d36a-openshift-default-dockercfg-6qxg6\"" Nov 01 22:39:52.810136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.781371 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-envoy\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.810136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.781453 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-certs\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.810136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.781541 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-credential-socket\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.810136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.781658 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-podinfo\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.810136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.781688 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05a5125-b946-4a15-b8e5-663a60ffba7c-istiod-ca-cert\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.810136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.781765 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-socket\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.810136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.781895 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-data\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.810136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.781987 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwsbj\" (UniqueName: \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-kube-api-access-jwsbj\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.810136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.782022 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-token\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.888130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.887154 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-podinfo\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.888130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.887209 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05a5125-b946-4a15-b8e5-663a60ffba7c-istiod-ca-cert\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.912132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.911247 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-socket\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.912132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.911414 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-data\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.912132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.911505 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwsbj\" (UniqueName: \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-kube-api-access-jwsbj\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.912132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.911542 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-token\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.912132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.911660 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-envoy\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.912132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.911728 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-certs\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.912132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.911831 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-credential-socket\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.918132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.914906 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-data\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.918132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.914965 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05a5125-b946-4a15-b8e5-663a60ffba7c-istiod-ca-cert\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.918132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.915194 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-socket\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.918132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.915479 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-credential-socket\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.919885 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.919767 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-certs\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.937081 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.937054 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-token\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.946265 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.946212 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwsbj\" (UniqueName: \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-kube-api-access-jwsbj\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.961857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.961828 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-envoy\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:52.984157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:52.981697 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-podinfo\") pod \"e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:53.184122 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:53.183812 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:53.618416 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:53.616554 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6"] Nov 01 22:39:53.621436 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:39:53.621385 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05a5125_b946_4a15_b8e5_663a60ffba7c.slice/crio-73dc3d6c0784bf4ff1fddd3f5c4e1f46b187f46b033c6fa6efa522b4c79c5faf WatchSource:0}: Error finding container 73dc3d6c0784bf4ff1fddd3f5c4e1f46b187f46b033c6fa6efa522b4c79c5faf: Status 404 returned error can't find the container with id 73dc3d6c0784bf4ff1fddd3f5c4e1f46b187f46b033c6fa6efa522b4c79c5faf Nov 01 22:39:53.624724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:53.624677 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:39:53.624855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:53.624781 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:39:53.624855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:53.624830 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:39:54.576801 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:54.576767 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" event={"ID":"d05a5125-b946-4a15-b8e5-663a60ffba7c","Type":"ContainerStarted","Data":"7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e"} Nov 01 22:39:54.577221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:54.576832 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" event={"ID":"d05a5125-b946-4a15-b8e5-663a60ffba7c","Type":"ContainerStarted","Data":"73dc3d6c0784bf4ff1fddd3f5c4e1f46b187f46b033c6fa6efa522b4c79c5faf"} Nov 01 22:39:54.606277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:54.606226 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" podStartSLOduration=3.6062120650000002 podStartE2EDuration="3.606212065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:54.601213073 +0000 UTC m=+998.385652007" watchObservedRunningTime="2026-11-01 22:39:54.606212065 +0000 UTC m=+998.390650985" Nov 01 22:39:55.187524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.187496 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:55.193569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.193544 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:55.605250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.605154 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:55.606284 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.606195 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:39:55.942605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.942582 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:39:55.992216 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.992180 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-certs\") pod \"80391b86-89a6-41db-8baa-59c62315596b\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " Nov 01 22:39:55.992384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.992246 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80391b86-89a6-41db-8baa-59c62315596b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80391b86-89a6-41db-8baa-59c62315596b-istio-podinfo\") pod \"80391b86-89a6-41db-8baa-59c62315596b\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " Nov 01 22:39:55.992384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.992289 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-istio-token\" (UniqueName: \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-istio-token\") pod \"80391b86-89a6-41db-8baa-59c62315596b\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " Nov 01 22:39:55.992526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.992431 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-data\") pod \"80391b86-89a6-41db-8baa-59c62315596b\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " Nov 01 22:39:55.992526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.992475 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-certs" pod "80391b86-89a6-41db-8baa-59c62315596b" (UID: "80391b86-89a6-41db-8baa-59c62315596b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:39:55.992634 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.992600 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-socket\") pod \"80391b86-89a6-41db-8baa-59c62315596b\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " Nov 01 22:39:55.992696 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.992640 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80391b86-89a6-41db-8baa-59c62315596b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80391b86-89a6-41db-8baa-59c62315596b-istiod-ca-cert\") pod \"80391b86-89a6-41db-8baa-59c62315596b\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " Nov 01 22:39:55.992753 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.992741 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-credential-socket\") pod \"80391b86-89a6-41db-8baa-59c62315596b\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " Nov 01 22:39:55.993068 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.992864 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-envoy\") pod \"80391b86-89a6-41db-8baa-59c62315596b\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " Nov 01 22:39:55.993068 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.992885 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-socket" pod "80391b86-89a6-41db-8baa-59c62315596b" (UID: "80391b86-89a6-41db-8baa-59c62315596b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:39:55.993068 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.992958 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-kube-api-access-mxdfk\" (UniqueName: \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-kube-api-access-mxdfk\") pod \"80391b86-89a6-41db-8baa-59c62315596b\" (UID: \"80391b86-89a6-41db-8baa-59c62315596b\") " Nov 01 22:39:55.993068 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.993031 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-credential-socket" pod "80391b86-89a6-41db-8baa-59c62315596b" (UID: "80391b86-89a6-41db-8baa-59c62315596b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:39:55.993350 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.993068 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80391b86-89a6-41db-8baa-59c62315596b-istiod-ca-cert" pod "80391b86-89a6-41db-8baa-59c62315596b" (UID: "80391b86-89a6-41db-8baa-59c62315596b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:55.993951 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.993924 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:55.994044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.993952 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:55.994044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.993969 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80391b86-89a6-41db-8baa-59c62315596b-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:55.994044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.993982 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:55.994254 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.994068 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-data" pod "80391b86-89a6-41db-8baa-59c62315596b" (UID: "80391b86-89a6-41db-8baa-59c62315596b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:39:55.995018 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.994994 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-istio-token" pod "80391b86-89a6-41db-8baa-59c62315596b" (UID: "80391b86-89a6-41db-8baa-59c62315596b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:55.995377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.995358 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80391b86-89a6-41db-8baa-59c62315596b-istio-podinfo" pod "80391b86-89a6-41db-8baa-59c62315596b" (UID: "80391b86-89a6-41db-8baa-59c62315596b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:39:55.996669 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.996645 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-envoy" pod "80391b86-89a6-41db-8baa-59c62315596b" (UID: "80391b86-89a6-41db-8baa-59c62315596b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:39:55.996787 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:55.996763 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-kube-api-access-mxdfk" pod "80391b86-89a6-41db-8baa-59c62315596b" (UID: "80391b86-89a6-41db-8baa-59c62315596b"). InnerVolumeSpecName "kube-api-access-mxdfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:56.095839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.095761 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:56.095839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.095792 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80391b86-89a6-41db-8baa-59c62315596b-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:56.095839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.095806 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxdfk\" (UniqueName: \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-kube-api-access-mxdfk\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:56.095839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.095821 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80391b86-89a6-41db-8baa-59c62315596b-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:56.095839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.095835 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80391b86-89a6-41db-8baa-59c62315596b-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:56.612321 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.612238 6466 generic.go:356] "Generic (PLEG): container finished" podID="80391b86-89a6-41db-8baa-59c62315596b" containerID="1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6" exitCode=0 Nov 01 22:39:56.612321 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.612284 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" Nov 01 22:39:56.612321 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.612309 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" event={"ID":"80391b86-89a6-41db-8baa-59c62315596b","Type":"ContainerDied","Data":"1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6"} Nov 01 22:39:56.612841 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.612360 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98" event={"ID":"80391b86-89a6-41db-8baa-59c62315596b","Type":"ContainerDied","Data":"aef6208e7add18b2b498aff0cff6d97f33bdb6147d98c18c3d4ce1a8d68aedf4"} Nov 01 22:39:56.612841 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.612383 6466 scope.go:122] "RemoveContainer" containerID="1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6" Nov 01 22:39:56.625701 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.625657 6466 scope.go:122] "RemoveContainer" containerID="4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8" Nov 01 22:39:56.641136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.641092 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98"] Nov 01 22:39:56.641272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.641160 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-66033bd2-openshift-default-5c6f796fb5-gjt98"] Nov 01 22:39:56.641420 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.641408 6466 scope.go:122] "RemoveContainer" containerID="1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6" Nov 01 22:39:56.641714 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:39:56.641697 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6\": container with ID starting with 1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6 not found: ID does not exist" containerID="1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6" Nov 01 22:39:56.641751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.641722 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6"} err="failed to get container status \"1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6\": rpc error: code = NotFound desc = could not find container \"1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6\": container with ID starting with 1be7a9d7d19ea5e550834070153b7a24d58c51970b29c4d3d3a03948a779a8e6 not found: ID does not exist" Nov 01 22:39:56.641751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.641741 6466 scope.go:122] "RemoveContainer" containerID="4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8" Nov 01 22:39:56.642011 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:39:56.641982 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8\": container with ID starting with 4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8 not found: ID does not exist" containerID="4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8" Nov 01 22:39:56.642165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:56.642013 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8"} err="failed to get container status \"4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8\": rpc error: code = NotFound desc = could not find container \"4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8\": container with ID starting with 4802e095a76fbc6cd5c3e9329f893d9988d5240ee3cd94599e4b88c499dc93b8 not found: ID does not exist" Nov 01 22:39:58.095860 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.095823 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75"] Nov 01 22:39:58.097167 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.097144 6466 cpu_manager.go:414] "containerMap: removing container" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" Nov 01 22:39:58.097246 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.097168 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" Nov 01 22:39:58.097246 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.097187 6466 cpu_manager.go:414] "containerMap: removing container" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" Nov 01 22:39:58.097246 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.097194 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" Nov 01 22:39:58.097453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.097439 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" Nov 01 22:39:58.103621 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.103596 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:39:58.110091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.110067 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75"] Nov 01 22:39:58.118422 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.117831 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-93d36a-dockercfg-k5sds\"" Nov 01 22:39:58.186131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.178426 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-93d36a\"" Nov 01 22:39:58.211479 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.202617 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xvcp\" (UniqueName: \"kubernetes.io/projected/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-kube-api-access-5xvcp\") pod \"payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75\" (UID: \"d9761117-3b6c-4b40-a45d-43d23dc0c9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:39:58.211479 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.202989 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75\" (UID: \"d9761117-3b6c-4b40-a45d-43d23dc0c9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:39:58.470560 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.448797 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xvcp\" (UniqueName: \"kubernetes.io/projected/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-kube-api-access-5xvcp\") pod \"payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75\" (UID: \"d9761117-3b6c-4b40-a45d-43d23dc0c9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:39:58.470560 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.449182 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75\" (UID: \"d9761117-3b6c-4b40-a45d-43d23dc0c9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:39:58.485311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.482574 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75\" (UID: \"d9761117-3b6c-4b40-a45d-43d23dc0c9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:39:58.505131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.504186 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xvcp\" (UniqueName: \"kubernetes.io/projected/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-kube-api-access-5xvcp\") pod \"payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75\" (UID: \"d9761117-3b6c-4b40-a45d-43d23dc0c9fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:39:58.617034 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.616992 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80391b86-89a6-41db-8baa-59c62315596b" path="/var/lib/kubelet/pods/80391b86-89a6-41db-8baa-59c62315596b/volumes" Nov 01 22:39:58.731857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:58.731757 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:39:59.091166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:59.091137 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75"] Nov 01 22:39:59.092334 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:39:59.092307 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9761117_3b6c_4b40_a45d_43d23dc0c9fd.slice/crio-646e8712673c66c846fb17040131377da8b9f4dc818400cc91f8fa944ad6031a WatchSource:0}: Error finding container 646e8712673c66c846fb17040131377da8b9f4dc818400cc91f8fa944ad6031a: Status 404 returned error can't find the container with id 646e8712673c66c846fb17040131377da8b9f4dc818400cc91f8fa944ad6031a Nov 01 22:39:59.754404 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:59.754365 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" event={"ID":"d9761117-3b6c-4b40-a45d-43d23dc0c9fd","Type":"ContainerStarted","Data":"df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865"} Nov 01 22:39:59.754813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:59.754421 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" event={"ID":"d9761117-3b6c-4b40-a45d-43d23dc0c9fd","Type":"ContainerStarted","Data":"646e8712673c66c846fb17040131377da8b9f4dc818400cc91f8fa944ad6031a"} Nov 01 22:39:59.754813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:59.754471 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:39:59.780699 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:39:59.780641 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" podStartSLOduration=1.780628397 podStartE2EDuration="1.780628397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:59.778313595 +0000 UTC m=+1003.562752516" watchObservedRunningTime="2026-11-01 22:39:59.780628397 +0000 UTC m=+1003.565067317" Nov 01 22:40:03.048187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.047927 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z"] Nov 01 22:40:03.049575 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.049541 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="80391b86-89a6-41db-8baa-59c62315596b" containerName="istio-proxy" Nov 01 22:40:03.054423 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.054392 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.073223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.061664 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-c9a7ce-openshift-default-dockercfg-lchkw\"" Nov 01 22:40:03.073223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.071368 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z"] Nov 01 22:40:03.100523 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.100479 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-socket\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.100727 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.100532 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101175a2-a477-494a-99ff-ac688c53c07e-istiod-ca-cert\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.100727 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.100708 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-data\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.100827 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.100810 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-envoy\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.100892 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.100871 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-credential-socket\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.101046 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.101030 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101175a2-a477-494a-99ff-ac688c53c07e-istio-podinfo\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.101171 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.101098 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-certs\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.101260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.101245 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzt95\" (UniqueName: \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-kube-api-access-rzt95\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.101323 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.101306 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-istio-token\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.224136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.223057 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-certs\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.224136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.223179 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzt95\" (UniqueName: \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-kube-api-access-rzt95\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.224136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.223240 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-istio-token\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.224136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.223301 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-socket\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.224136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.223320 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101175a2-a477-494a-99ff-ac688c53c07e-istiod-ca-cert\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.224136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.223396 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-data\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.224136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.223465 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-envoy\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.224136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.223511 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-credential-socket\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.224136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.223624 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101175a2-a477-494a-99ff-ac688c53c07e-istio-podinfo\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.274127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.274059 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-data\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.274712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.274680 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-socket\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.277899 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.275666 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101175a2-a477-494a-99ff-ac688c53c07e-istiod-ca-cert\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.301252 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.285576 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-certs\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.366132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.315560 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-credential-socket\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.499579 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.492439 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-envoy\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.564752 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.549933 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101175a2-a477-494a-99ff-ac688c53c07e-istio-podinfo\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.605443 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.605401 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-istio-token\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.636192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.635549 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzt95\" (UniqueName: \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-kube-api-access-rzt95\") pod \"e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:03.693710 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:03.693534 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:04.093297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:04.093262 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z"] Nov 01 22:40:04.093855 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:40:04.093810 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101175a2_a477_494a_99ff_ac688c53c07e.slice/crio-1a50b73399d1d7bbe59f34c3b91dd74ddcee038763b601281f5aa3692d65da7d WatchSource:0}: Error finding container 1a50b73399d1d7bbe59f34c3b91dd74ddcee038763b601281f5aa3692d65da7d: Status 404 returned error can't find the container with id 1a50b73399d1d7bbe59f34c3b91dd74ddcee038763b601281f5aa3692d65da7d Nov 01 22:40:04.097779 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:04.097735 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:40:04.097868 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:04.097843 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:40:04.097903 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:04.097891 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:40:04.820862 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:04.820828 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" event={"ID":"101175a2-a477-494a-99ff-ac688c53c07e","Type":"ContainerStarted","Data":"b01b22ec0d0d3a6a351b46ad2b60a5f7a70ee174fe1cbe3fcd146eca65e4e4d6"} Nov 01 22:40:04.821044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:04.820883 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" event={"ID":"101175a2-a477-494a-99ff-ac688c53c07e","Type":"ContainerStarted","Data":"1a50b73399d1d7bbe59f34c3b91dd74ddcee038763b601281f5aa3692d65da7d"} Nov 01 22:40:04.847370 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:04.847322 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" podStartSLOduration=1.8473070360000001 podStartE2EDuration="1.847307036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:04.842355672 +0000 UTC m=+1008.626794604" watchObservedRunningTime="2026-11-01 22:40:04.847307036 +0000 UTC m=+1008.631745956" Nov 01 22:40:05.694019 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:05.693991 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:05.699797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:05.699775 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:05.828974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:05.828942 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:05.829881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:05.829862 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:40:07.075755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.075723 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk"] Nov 01 22:40:07.081445 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.081408 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.088371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.088339 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-93d36a-serving-cert\"" Nov 01 22:40:07.094342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.094306 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk"] Nov 01 22:40:07.215254 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.205605 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.227853 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.215586 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzgtz\" (UniqueName: \"kubernetes.io/projected/1fed40e3-422e-49b2-b12b-b92e6c53c174-kube-api-access-xzgtz\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.228305 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.228274 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.228757 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.228735 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fed40e3-422e-49b2-b12b-b92e6c53c174-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.332175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.330565 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzgtz\" (UniqueName: \"kubernetes.io/projected/1fed40e3-422e-49b2-b12b-b92e6c53c174-kube-api-access-xzgtz\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.332175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.330753 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.332175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.331044 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fed40e3-422e-49b2-b12b-b92e6c53c174-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.332175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.331177 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.336816 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.334058 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fed40e3-422e-49b2-b12b-b92e6c53c174-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.336816 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.336200 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.342797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.342770 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.355899 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.355814 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzgtz\" (UniqueName: \"kubernetes.io/projected/1fed40e3-422e-49b2-b12b-b92e6c53c174-kube-api-access-xzgtz\") pod \"maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.406013 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.405978 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:07.699561 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:40:07.699525 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fed40e3_422e_49b2_b12b_b92e6c53c174.slice/crio-0e0ca3ab01aad2e8bac7738e4cab6a24efb9fc34e6ae0049da42dda6505aecad WatchSource:0}: Error finding container 0e0ca3ab01aad2e8bac7738e4cab6a24efb9fc34e6ae0049da42dda6505aecad: Status 404 returned error can't find the container with id 0e0ca3ab01aad2e8bac7738e4cab6a24efb9fc34e6ae0049da42dda6505aecad Nov 01 22:40:07.699721 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.699603 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk"] Nov 01 22:40:07.847090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:07.847052 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" event={"ID":"1fed40e3-422e-49b2-b12b-b92e6c53c174","Type":"ContainerStarted","Data":"0e0ca3ab01aad2e8bac7738e4cab6a24efb9fc34e6ae0049da42dda6505aecad"} Nov 01 22:40:08.858461 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:08.858421 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" event={"ID":"1fed40e3-422e-49b2-b12b-b92e6c53c174","Type":"ContainerStarted","Data":"8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91"} Nov 01 22:40:08.858962 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:08.858499 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:08.894639 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:08.894581 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" podStartSLOduration=1.310995317 podStartE2EDuration="1.894564591s" totalImagesPullingTime="583.569274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:08.88769474 +0000 UTC m=+1012.672133662" watchObservedRunningTime="2026-11-01 22:40:08.894564591 +0000 UTC m=+1012.679003541" Nov 01 22:40:10.781157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:10.777914 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:40:14.872471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:14.872445 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:40:19.282765 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:40:19.282725 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:40:28.529199 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:28.529163 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d"] Nov 01 22:40:28.559134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:28.558390 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:40:28.568232 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:28.568199 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf"] Nov 01 22:40:28.608144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:28.605393 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-dcd5d4\"" Nov 01 22:40:28.615633 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:28.613468 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-dcd5d4-dockercfg-d5tsh\"" Nov 01 22:40:28.723132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:28.704989 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:40:28.790426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:28.724598 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckcq9\" (UniqueName: \"kubernetes.io/projected/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-kube-api-access-ckcq9\") pod \"payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d\" (UID: \"2b61ca43-37d4-4a9d-a4c7-0ee73742f505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:40:28.816951 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:28.808483 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d\" (UID: \"2b61ca43-37d4-4a9d-a4c7-0ee73742f505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:40:28.986703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:28.986669 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d"] Nov 01 22:40:29.139364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.136236 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqk6n\" (UniqueName: \"kubernetes.io/projected/ddc44afb-2487-41d2-8883-1ae6736168b2-kube-api-access-sqk6n\") pod \"payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf\" (UID: \"ddc44afb-2487-41d2-8883-1ae6736168b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:40:29.168337 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.168301 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc44afb-2487-41d2-8883-1ae6736168b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf\" (UID: \"ddc44afb-2487-41d2-8883-1ae6736168b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:40:29.168821 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.168786 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckcq9\" (UniqueName: \"kubernetes.io/projected/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-kube-api-access-ckcq9\") pod \"payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d\" (UID: \"2b61ca43-37d4-4a9d-a4c7-0ee73742f505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:40:29.281237 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.273253 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d\" (UID: \"2b61ca43-37d4-4a9d-a4c7-0ee73742f505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:40:29.281237 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.274605 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d\" (UID: \"2b61ca43-37d4-4a9d-a4c7-0ee73742f505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:40:29.327987 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.327952 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf"] Nov 01 22:40:29.379945 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.379899 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqk6n\" (UniqueName: \"kubernetes.io/projected/ddc44afb-2487-41d2-8883-1ae6736168b2-kube-api-access-sqk6n\") pod \"payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf\" (UID: \"ddc44afb-2487-41d2-8883-1ae6736168b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:40:29.380349 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.380325 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc44afb-2487-41d2-8883-1ae6736168b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf\" (UID: \"ddc44afb-2487-41d2-8883-1ae6736168b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:40:29.381593 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.381568 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc44afb-2487-41d2-8883-1ae6736168b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf\" (UID: \"ddc44afb-2487-41d2-8883-1ae6736168b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:40:29.444413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.444328 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqk6n\" (UniqueName: \"kubernetes.io/projected/ddc44afb-2487-41d2-8883-1ae6736168b2-kube-api-access-sqk6n\") pod \"payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf\" (UID: \"ddc44afb-2487-41d2-8883-1ae6736168b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:40:29.480904 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.444742 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckcq9\" (UniqueName: \"kubernetes.io/projected/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-kube-api-access-ckcq9\") pod \"payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d\" (UID: \"2b61ca43-37d4-4a9d-a4c7-0ee73742f505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:40:29.486709 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.485614 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:40:29.577060 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:29.577025 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:40:30.894513 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:30.894477 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf"] Nov 01 22:40:30.895683 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:40:30.895636 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc44afb_2487_41d2_8883_1ae6736168b2.slice/crio-ced3ff1f5a01a9bdd4f0f311e5a977ce551b8204f5958fd95999fb53c85dded1 WatchSource:0}: Error finding container ced3ff1f5a01a9bdd4f0f311e5a977ce551b8204f5958fd95999fb53c85dded1: Status 404 returned error can't find the container with id ced3ff1f5a01a9bdd4f0f311e5a977ce551b8204f5958fd95999fb53c85dded1 Nov 01 22:40:30.901225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:30.901198 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d"] Nov 01 22:40:30.906245 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:40:30.906222 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b61ca43_37d4_4a9d_a4c7_0ee73742f505.slice/crio-19651f9d03be34b4c04c25ea4c2b233c8a04ca551ca059bcd08c90f8427b255e WatchSource:0}: Error finding container 19651f9d03be34b4c04c25ea4c2b233c8a04ca551ca059bcd08c90f8427b255e: Status 404 returned error can't find the container with id 19651f9d03be34b4c04c25ea4c2b233c8a04ca551ca059bcd08c90f8427b255e Nov 01 22:40:31.614014 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:31.613970 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" event={"ID":"ddc44afb-2487-41d2-8883-1ae6736168b2","Type":"ContainerStarted","Data":"d7eb587bb0e2998fff64469867c3ff172cd6a0e3c422362efa2b0f8636ed9f33"} Nov 01 22:40:31.614239 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:31.614031 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" event={"ID":"ddc44afb-2487-41d2-8883-1ae6736168b2","Type":"ContainerStarted","Data":"ced3ff1f5a01a9bdd4f0f311e5a977ce551b8204f5958fd95999fb53c85dded1"} Nov 01 22:40:31.614239 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:31.614061 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:40:31.615690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:31.615662 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" event={"ID":"2b61ca43-37d4-4a9d-a4c7-0ee73742f505","Type":"ContainerStarted","Data":"4d515b262943b333a1c7e4460669e5892204de5ca94377dac60c8b50868a7f51"} Nov 01 22:40:31.615815 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:31.615710 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" event={"ID":"2b61ca43-37d4-4a9d-a4c7-0ee73742f505","Type":"ContainerStarted","Data":"19651f9d03be34b4c04c25ea4c2b233c8a04ca551ca059bcd08c90f8427b255e"} Nov 01 22:40:31.615815 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:31.615763 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:40:31.662307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:31.662164 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" podStartSLOduration=3.662093302 podStartE2EDuration="3.662093302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:31.660490906 +0000 UTC m=+1035.444929838" watchObservedRunningTime="2026-11-01 22:40:31.662093302 +0000 UTC m=+1035.446532225" Nov 01 22:40:31.682521 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:31.682471 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" podStartSLOduration=3.682458117 podStartE2EDuration="3.682458117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:31.679729005 +0000 UTC m=+1035.464167939" watchObservedRunningTime="2026-11-01 22:40:31.682458117 +0000 UTC m=+1035.466897037" Nov 01 22:40:35.607222 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:35.607186 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7021-2bc28e"] Nov 01 22:40:35.612920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:35.612897 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7021-2bc28e" Nov 01 22:40:35.621159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:35.619283 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vrfk9\"" Nov 01 22:40:35.621159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:35.620986 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7021-2bc28e"] Nov 01 22:40:35.741830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:35.741801 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kll6l\" (UniqueName: \"kubernetes.io/projected/dee31e73-6791-4acf-96c0-ea26a284d50a-kube-api-access-kll6l\") pod \"test-curl-7021-2bc28e\" (UID: \"dee31e73-6791-4acf-96c0-ea26a284d50a\") " pod="opendatahub/test-curl-7021-2bc28e" Nov 01 22:40:35.843835 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:35.843799 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kll6l\" (UniqueName: \"kubernetes.io/projected/dee31e73-6791-4acf-96c0-ea26a284d50a-kube-api-access-kll6l\") pod \"test-curl-7021-2bc28e\" (UID: \"dee31e73-6791-4acf-96c0-ea26a284d50a\") " pod="opendatahub/test-curl-7021-2bc28e" Nov 01 22:40:35.852157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:35.852130 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kll6l\" (UniqueName: \"kubernetes.io/projected/dee31e73-6791-4acf-96c0-ea26a284d50a-kube-api-access-kll6l\") pod \"test-curl-7021-2bc28e\" (UID: \"dee31e73-6791-4acf-96c0-ea26a284d50a\") " pod="opendatahub/test-curl-7021-2bc28e" Nov 01 22:40:35.932599 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:35.932568 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7021-2bc28e" Nov 01 22:40:36.301786 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:36.282647 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7021-2bc28e"] Nov 01 22:40:36.305186 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:40:36.304073 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddee31e73_6791_4acf_96c0_ea26a284d50a.slice/crio-65e0bf5aa3860071ace3e162bac948928904da142f4aad76d94d99cc4e86a52c WatchSource:0}: Error finding container 65e0bf5aa3860071ace3e162bac948928904da142f4aad76d94d99cc4e86a52c: Status 404 returned error can't find the container with id 65e0bf5aa3860071ace3e162bac948928904da142f4aad76d94d99cc4e86a52c Nov 01 22:40:36.666916 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:36.666882 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7021-2bc28e" event={"ID":"dee31e73-6791-4acf-96c0-ea26a284d50a","Type":"ContainerStarted","Data":"72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad"} Nov 01 22:40:36.667393 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:36.666932 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7021-2bc28e" event={"ID":"dee31e73-6791-4acf-96c0-ea26a284d50a","Type":"ContainerStarted","Data":"65e0bf5aa3860071ace3e162bac948928904da142f4aad76d94d99cc4e86a52c"} Nov 01 22:40:36.684087 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:36.684039 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7021-2bc28e" podStartSLOduration=1.684024381 podStartE2EDuration="1.684024381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:36.68288187 +0000 UTC m=+1040.467320791" watchObservedRunningTime="2026-11-01 22:40:36.684024381 +0000 UTC m=+1040.468463301" Nov 01 22:40:37.505131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:37.501168 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7021-2bc28e"] Nov 01 22:40:37.522835 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:37.522421 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7021-2bc28e"] Nov 01 22:40:38.766150 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:38.765820 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7021-2bc28e" podUID="dee31e73-6791-4acf-96c0-ea26a284d50a" containerName="test-curl-7021-2bc28e" containerID="cri-o://72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad" gracePeriod=2 Nov 01 22:40:40.606492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:40.606458 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:40:40.606929 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:40.606524 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 22:40:40.787317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:40.787288 6466 generic.go:356] "Generic (PLEG): container finished" podID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerID="7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e" exitCode=0 Nov 01 22:40:40.787444 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:40.787363 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" event={"ID":"d05a5125-b946-4a15-b8e5-663a60ffba7c","Type":"ContainerDied","Data":"7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e"} Nov 01 22:40:40.789027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:40.788502 6466 scope.go:122] "RemoveContainer" containerID="7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e" Nov 01 22:40:40.793399 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:40.793358 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:40:40.793539 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:40.793518 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:40:40.793635 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:40.793618 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:40:41.154754 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.154728 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7021-2bc28e" Nov 01 22:40:41.273447 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.273412 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee31e73-6791-4acf-96c0-ea26a284d50a-kube-api-access-kll6l\" (UniqueName: \"kubernetes.io/projected/dee31e73-6791-4acf-96c0-ea26a284d50a-kube-api-access-kll6l\") pod \"dee31e73-6791-4acf-96c0-ea26a284d50a\" (UID: \"dee31e73-6791-4acf-96c0-ea26a284d50a\") " Nov 01 22:40:41.276493 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.276451 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee31e73-6791-4acf-96c0-ea26a284d50a-kube-api-access-kll6l" pod "dee31e73-6791-4acf-96c0-ea26a284d50a" (UID: "dee31e73-6791-4acf-96c0-ea26a284d50a"). InnerVolumeSpecName "kube-api-access-kll6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:41.376242 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.376205 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kll6l\" (UniqueName: \"kubernetes.io/projected/dee31e73-6791-4acf-96c0-ea26a284d50a-kube-api-access-kll6l\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:41.790227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.790194 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74"] Nov 01 22:40:41.791310 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.791285 6466 cpu_manager.go:414] "containerMap: removing container" podUID="dee31e73-6791-4acf-96c0-ea26a284d50a" containerName="test-curl-7021-2bc28e" Nov 01 22:40:41.791310 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.791308 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee31e73-6791-4acf-96c0-ea26a284d50a" containerName="test-curl-7021-2bc28e" Nov 01 22:40:41.791558 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.791544 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee31e73-6791-4acf-96c0-ea26a284d50a" containerName="test-curl-7021-2bc28e" Nov 01 22:40:41.794432 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.794411 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:41.797318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.797294 6466 generic.go:356] "Generic (PLEG): container finished" podID="dee31e73-6791-4acf-96c0-ea26a284d50a" containerID="72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad" exitCode=137 Nov 01 22:40:41.797423 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.797344 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7021-2bc28e" Nov 01 22:40:41.797423 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.797390 6466 scope.go:122] "RemoveContainer" containerID="72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad" Nov 01 22:40:41.804009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.803971 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" event={"ID":"d05a5125-b946-4a15-b8e5-663a60ffba7c","Type":"ContainerStarted","Data":"150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b"} Nov 01 22:40:41.806751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.806183 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74"] Nov 01 22:40:41.806751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.806267 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-dcd5d4-serving-cert\"" Nov 01 22:40:41.816199 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.815485 6466 scope.go:122] "RemoveContainer" containerID="72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad" Nov 01 22:40:41.816199 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:40:41.816025 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad\": container with ID starting with 72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad not found: ID does not exist" containerID="72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad" Nov 01 22:40:41.816199 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.816117 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad"} err="failed to get container status \"72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad\": rpc error: code = NotFound desc = could not find container \"72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad\": container with ID starting with 72e3e9197b092d01d47f8ce5c3854a6d9e01154629e66c6a5c11629f01c148ad not found: ID does not exist" Nov 01 22:40:41.852205 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:40:41.852154 6466 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7021-2bc28e\" is forbidden: User \"system:node:ip-10-0-134-88.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-88.us-east-2.compute.internal' and this object" podUID="dee31e73-6791-4acf-96c0-ea26a284d50a" pod="opendatahub/test-curl-7021-2bc28e" Nov 01 22:40:41.886556 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.886515 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:41.886926 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.886899 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:41.887091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.886947 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/875665f0-e7d1-49bc-af44-600e5c8e9ce1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:41.887091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:41.887068 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dj26\" (UniqueName: \"kubernetes.io/projected/875665f0-e7d1-49bc-af44-600e5c8e9ce1-kube-api-access-7dj26\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:42.041257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.003703 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:42.041588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.041554 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/875665f0-e7d1-49bc-af44-600e5c8e9ce1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:42.041922 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.041901 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dj26\" (UniqueName: \"kubernetes.io/projected/875665f0-e7d1-49bc-af44-600e5c8e9ce1-kube-api-access-7dj26\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:42.042088 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.042069 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:42.043043 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.042718 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/875665f0-e7d1-49bc-af44-600e5c8e9ce1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:42.043279 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:40:42.012292 6466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-serving-cert: secret "maas-api-e2e-shared-b-w5-dcd5d4-serving-cert" not found Nov 01 22:40:42.043485 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:40:42.043471 6466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-maas-api-tls podName:875665f0-e7d1-49bc-af44-600e5c8e9ce1 nodeName:}" failed. No retries permitted until 2026-11-01 22:40:42.543425485 +0000 UTC m=+1046.327864385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-maas-api-tls") pod "maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" (UID: "875665f0-e7d1-49bc-af44-600e5c8e9ce1") : secret "maas-api-e2e-shared-b-w5-dcd5d4-serving-cert" not found Nov 01 22:40:42.100757 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.100714 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:42.125046 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.124967 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dj26\" (UniqueName: \"kubernetes.io/projected/875665f0-e7d1-49bc-af44-600e5c8e9ce1-kube-api-access-7dj26\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:42.195205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.190537 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:40:42.195205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.190661 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:40:42.195205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.190703 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 22:40:42.367730 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.367641 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee31e73-6791-4acf-96c0-ea26a284d50a" path="/var/lib/kubelet/pods/dee31e73-6791-4acf-96c0-ea26a284d50a/volumes" Nov 01 22:40:42.573156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.572779 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:42.620244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.620084 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:42.664136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.663412 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:40:42.680208 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.665667 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:40:42.724834 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:42.724801 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:43.245129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:43.230590 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:40:43.248439 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:43.248402 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 22:40:43.885456 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:43.884453 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74"] Nov 01 22:40:43.977995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:43.977959 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt"] Nov 01 22:40:43.985163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:43.985134 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:40:43.987874 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:43.987848 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt"] Nov 01 22:40:44.037131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.031837 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-3fa400\"" Nov 01 22:40:44.089130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.057522 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-3fa400-dockercfg-fph9z\"" Nov 01 22:40:44.177131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.171689 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3092930b-ddb0-4915-8967-34ef095ae412-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt\" (UID: \"3092930b-ddb0-4915-8967-34ef095ae412\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:40:44.177131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.171769 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktqfr\" (UniqueName: \"kubernetes.io/projected/3092930b-ddb0-4915-8967-34ef095ae412-kube-api-access-ktqfr\") pod \"payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt\" (UID: \"3092930b-ddb0-4915-8967-34ef095ae412\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:40:44.231131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.230621 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:40:44.231131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.230705 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 22:40:44.280823 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.280776 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3092930b-ddb0-4915-8967-34ef095ae412-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt\" (UID: \"3092930b-ddb0-4915-8967-34ef095ae412\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:40:44.281399 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.281377 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqfr\" (UniqueName: \"kubernetes.io/projected/3092930b-ddb0-4915-8967-34ef095ae412-kube-api-access-ktqfr\") pod \"payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt\" (UID: \"3092930b-ddb0-4915-8967-34ef095ae412\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:40:44.286131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.282820 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3092930b-ddb0-4915-8967-34ef095ae412-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt\" (UID: \"3092930b-ddb0-4915-8967-34ef095ae412\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:40:44.374002 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.373970 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktqfr\" (UniqueName: \"kubernetes.io/projected/3092930b-ddb0-4915-8967-34ef095ae412-kube-api-access-ktqfr\") pod \"payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt\" (UID: \"3092930b-ddb0-4915-8967-34ef095ae412\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:40:44.475902 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.475867 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" event={"ID":"875665f0-e7d1-49bc-af44-600e5c8e9ce1","Type":"ContainerStarted","Data":"e61cf950067879015ccd2cc5389aa4c13ae2f71ecf99757ff4859d772ffe0098"} Nov 01 22:40:44.601475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:44.601448 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:40:45.132988 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:40:45.132945 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3092930b_ddb0_4915_8967_34ef095ae412.slice/crio-5f381d647c6a9c86f3f4c8a5cb81e4af10a0f77bcb1cb2d1fd89b8d3bfe62c8e WatchSource:0}: Error finding container 5f381d647c6a9c86f3f4c8a5cb81e4af10a0f77bcb1cb2d1fd89b8d3bfe62c8e: Status 404 returned error can't find the container with id 5f381d647c6a9c86f3f4c8a5cb81e4af10a0f77bcb1cb2d1fd89b8d3bfe62c8e Nov 01 22:40:45.133180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:45.133015 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt"] Nov 01 22:40:45.188241 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:45.188208 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:40:45.188375 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:45.188281 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 22:40:45.546827 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:45.545385 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" event={"ID":"3092930b-ddb0-4915-8967-34ef095ae412","Type":"ContainerStarted","Data":"d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8"} Nov 01 22:40:45.546827 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:45.545453 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" event={"ID":"3092930b-ddb0-4915-8967-34ef095ae412","Type":"ContainerStarted","Data":"5f381d647c6a9c86f3f4c8a5cb81e4af10a0f77bcb1cb2d1fd89b8d3bfe62c8e"} Nov 01 22:40:45.546827 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:45.546715 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:40:45.564167 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:45.563240 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" event={"ID":"875665f0-e7d1-49bc-af44-600e5c8e9ce1","Type":"ContainerStarted","Data":"e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e"} Nov 01 22:40:45.565075 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:45.565011 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:45.743164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:45.716241 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" podStartSLOduration=4.139697969 podStartE2EDuration="4.716220214s" totalImagesPullingTime="576.522245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:45.671083009 +0000 UTC m=+1049.455521932" watchObservedRunningTime="2026-11-01 22:40:45.716220214 +0000 UTC m=+1049.500659135" Nov 01 22:40:45.789348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:45.789285 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" podStartSLOduration=2.789267423 podStartE2EDuration="2.789267423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:45.594948893 +0000 UTC m=+1049.379387814" watchObservedRunningTime="2026-11-01 22:40:45.789267423 +0000 UTC m=+1049.573706344" Nov 01 22:40:46.187874 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:46.187838 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:40:46.188063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:46.187890 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 22:40:47.194132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:47.187796 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:40:47.194132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:47.187862 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 22:40:48.162595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:48.162563 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9"] Nov 01 22:40:48.187887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:48.187852 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:40:48.188080 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:48.187909 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 22:40:49.198132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:49.194778 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:40:49.198132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:49.196971 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:40:49.198132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:49.197992 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:40:49.295188 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:40:49.295140 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:40:52.627033 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:52.627005 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:40:53.163027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:53.162986 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" podUID="ee812bbc-878f-4308-a1dc-e277443bcbcf" containerName="payload-processing" containerID="cri-o://5a90e98cb3cd17ec81ac4de3a8c3bff6bdc23aa9f95854a0fd321f3fde71b7d6" gracePeriod=25 Nov 01 22:40:57.634299 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:40:57.632209 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:41:00.248463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:00.248425 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" podUID="ee812bbc-878f-4308-a1dc-e277443bcbcf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:41:09.981221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:09.953267 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:41:10.100165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:10.077177 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g"] Nov 01 22:41:10.189418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:10.144667 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:41:10.358182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:10.357338 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g"] Nov 01 22:41:10.405209 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:10.365292 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:10.547393 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:10.547353 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" podUID="ee812bbc-878f-4308-a1dc-e277443bcbcf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:41:10.599289 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:10.599238 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:41:10.607144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:10.603708 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b4f6bb-openshift-default-dockercfg-rjq5b\"" Nov 01 22:41:11.384269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.321609 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.514935 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.469515 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.685186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.676222 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-istio-token\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:11.685186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.676500 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltjdx\" (UniqueName: \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-kube-api-access-ltjdx\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:11.685186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.676558 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-socket\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:11.685186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.676626 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-envoy\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:11.685186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.676754 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-certs\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:11.685186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.676820 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e34de96-6bf5-411e-b72d-2010871edf08-istiod-ca-cert\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:11.697696 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.697655 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e34de96-6bf5-411e-b72d-2010871edf08-istio-podinfo\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:11.703326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.703284 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-credential-socket\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:11.703633 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.703613 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-data\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:11.953179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.947291 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.953179 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:41:11.952947 6466 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.596s" Nov 01 22:41:11.953179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.953027 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:41:11.953179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.953158 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh"] Nov 01 22:41:11.980132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.972361 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:41:11.999132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:11.992880 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:41:12.023132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.017203 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ee8d1a-dockercfg-7qtxc\"" Nov 01 22:41:12.032178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.029436 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ee8d1a\"" Nov 01 22:41:12.032178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.031249 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-istio-token\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.032178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.031381 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltjdx\" (UniqueName: \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-kube-api-access-ltjdx\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.032178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.031419 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-socket\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.032178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.031464 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-envoy\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.032178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.031509 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-certs\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.032178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.031562 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e34de96-6bf5-411e-b72d-2010871edf08-istiod-ca-cert\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.032178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.031604 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e34de96-6bf5-411e-b72d-2010871edf08-istio-podinfo\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.032178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.031655 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-credential-socket\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.032178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.031714 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-data\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.032813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.032286 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-data\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.057975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.057582 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-socket\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.105132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.076409 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-envoy\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.105132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.076728 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-certs\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.175383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.174574 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e34de96-6bf5-411e-b72d-2010871edf08-istiod-ca-cert\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.284189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.276280 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh"] Nov 01 22:41:12.309163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.290906 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-istio-token\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.309163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.292841 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:41:12.357132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.348402 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltjdx\" (UniqueName: \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-kube-api-access-ltjdx\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.375130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.370537 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-credential-socket\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.375130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.371193 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e34de96-6bf5-411e-b72d-2010871edf08-istio-podinfo\") pod \"e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.393487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.386226 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556faa39-b98e-4133-adea-bce7ed14751a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh\" (UID: \"556faa39-b98e-4133-adea-bce7ed14751a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:41:12.393487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.386279 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g757s\" (UniqueName: \"kubernetes.io/projected/556faa39-b98e-4133-adea-bce7ed14751a-kube-api-access-g757s\") pod \"payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh\" (UID: \"556faa39-b98e-4133-adea-bce7ed14751a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:41:12.529397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.529351 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:41:12.546263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.541085 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556faa39-b98e-4133-adea-bce7ed14751a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh\" (UID: \"556faa39-b98e-4133-adea-bce7ed14751a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:41:12.546263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.541191 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g757s\" (UniqueName: \"kubernetes.io/projected/556faa39-b98e-4133-adea-bce7ed14751a-kube-api-access-g757s\") pod \"payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh\" (UID: \"556faa39-b98e-4133-adea-bce7ed14751a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:41:12.546263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.543666 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556faa39-b98e-4133-adea-bce7ed14751a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh\" (UID: \"556faa39-b98e-4133-adea-bce7ed14751a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:41:12.591156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.547940 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:12.617318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.616342 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g757s\" (UniqueName: \"kubernetes.io/projected/556faa39-b98e-4133-adea-bce7ed14751a-kube-api-access-g757s\") pod \"payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh\" (UID: \"556faa39-b98e-4133-adea-bce7ed14751a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:41:12.688132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.678706 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" podUID="66937625-7220-44c9-b11a-dbc627d0219c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 01 22:41:12.727203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.725766 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:41:12.898257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.898229 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:41:12.989134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.963261 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:41:12.989134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:12.972423 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:41:13.022422 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:13.022394 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:41:13.684103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:13.682828 6466 generic.go:356] "Generic (PLEG): container finished" podID="ee812bbc-878f-4308-a1dc-e277443bcbcf" containerID="5a90e98cb3cd17ec81ac4de3a8c3bff6bdc23aa9f95854a0fd321f3fde71b7d6" exitCode=0 Nov 01 22:41:13.684103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:13.683360 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" event={"ID":"ee812bbc-878f-4308-a1dc-e277443bcbcf","Type":"ContainerDied","Data":"5a90e98cb3cd17ec81ac4de3a8c3bff6bdc23aa9f95854a0fd321f3fde71b7d6"} Nov 01 22:41:14.015827 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.015801 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh"] Nov 01 22:41:14.028149 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:41:14.028099 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556faa39_b98e_4133_adea_bce7ed14751a.slice/crio-f7440076c8a24f150127a3768e7cc24261447d75a0a763dd23c95c2afbe5b4c4 WatchSource:0}: Error finding container f7440076c8a24f150127a3768e7cc24261447d75a0a763dd23c95c2afbe5b4c4: Status 404 returned error can't find the container with id f7440076c8a24f150127a3768e7cc24261447d75a0a763dd23c95c2afbe5b4c4 Nov 01 22:41:14.068239 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.068212 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g"] Nov 01 22:41:14.069661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.069640 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:41:14.078142 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:41:14.078085 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e34de96_6bf5_411e_b72d_2010871edf08.slice/crio-be0f5bcc9e828af970ae29efdf7c920ca232169d0b9a4f33f2bf53fe7f172861 WatchSource:0}: Error finding container be0f5bcc9e828af970ae29efdf7c920ca232169d0b9a4f33f2bf53fe7f172861: Status 404 returned error can't find the container with id be0f5bcc9e828af970ae29efdf7c920ca232169d0b9a4f33f2bf53fe7f172861 Nov 01 22:41:14.081787 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.081748 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:41:14.081862 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.081849 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:41:14.081907 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.081895 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:41:14.181526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.181222 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee812bbc-878f-4308-a1dc-e277443bcbcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee812bbc-878f-4308-a1dc-e277443bcbcf-plugins-config-volume\") pod \"ee812bbc-878f-4308-a1dc-e277443bcbcf\" (UID: \"ee812bbc-878f-4308-a1dc-e277443bcbcf\") " Nov 01 22:41:14.182584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.181735 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee812bbc-878f-4308-a1dc-e277443bcbcf-kube-api-access-6lhnc\" (UniqueName: \"kubernetes.io/projected/ee812bbc-878f-4308-a1dc-e277443bcbcf-kube-api-access-6lhnc\") pod \"ee812bbc-878f-4308-a1dc-e277443bcbcf\" (UID: \"ee812bbc-878f-4308-a1dc-e277443bcbcf\") " Nov 01 22:41:14.182584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.182054 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee812bbc-878f-4308-a1dc-e277443bcbcf-plugins-config-volume" pod "ee812bbc-878f-4308-a1dc-e277443bcbcf" (UID: "ee812bbc-878f-4308-a1dc-e277443bcbcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:14.182584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.182560 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee812bbc-878f-4308-a1dc-e277443bcbcf-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:14.187027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.187000 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee812bbc-878f-4308-a1dc-e277443bcbcf-kube-api-access-6lhnc" pod "ee812bbc-878f-4308-a1dc-e277443bcbcf" (UID: "ee812bbc-878f-4308-a1dc-e277443bcbcf"). InnerVolumeSpecName "kube-api-access-6lhnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:14.284417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.284379 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lhnc\" (UniqueName: \"kubernetes.io/projected/ee812bbc-878f-4308-a1dc-e277443bcbcf-kube-api-access-6lhnc\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:14.787866 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.787826 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" event={"ID":"556faa39-b98e-4133-adea-bce7ed14751a","Type":"ContainerStarted","Data":"2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338"} Nov 01 22:41:14.788465 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.788436 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" event={"ID":"556faa39-b98e-4133-adea-bce7ed14751a","Type":"ContainerStarted","Data":"f7440076c8a24f150127a3768e7cc24261447d75a0a763dd23c95c2afbe5b4c4"} Nov 01 22:41:14.800509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.793358 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:41:14.806289 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.806257 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" event={"ID":"ee812bbc-878f-4308-a1dc-e277443bcbcf","Type":"ContainerDied","Data":"6b636773dfe77baf7fb48f55c02ee1cf4ca06537c8688bd9d623fef8f502cf73"} Nov 01 22:41:14.810803 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.810779 6466 scope.go:122] "RemoveContainer" containerID="5a90e98cb3cd17ec81ac4de3a8c3bff6bdc23aa9f95854a0fd321f3fde71b7d6" Nov 01 22:41:14.811101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.807140 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9" Nov 01 22:41:14.823740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.811516 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" event={"ID":"3e34de96-6bf5-411e-b72d-2010871edf08","Type":"ContainerStarted","Data":"f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360"} Nov 01 22:41:14.824178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.824144 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" event={"ID":"3e34de96-6bf5-411e-b72d-2010871edf08","Type":"ContainerStarted","Data":"be0f5bcc9e828af970ae29efdf7c920ca232169d0b9a4f33f2bf53fe7f172861"} Nov 01 22:41:14.827612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.827560 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" podStartSLOduration=4.827543627 podStartE2EDuration="4.827543627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:14.82476523 +0000 UTC m=+1078.609204151" watchObservedRunningTime="2026-11-01 22:41:14.827543627 +0000 UTC m=+1078.611982550" Nov 01 22:41:14.967575 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:14.967510 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podStartSLOduration=5.967484989 podStartE2EDuration="5.967484989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:14.857788429 +0000 UTC m=+1078.642227351" watchObservedRunningTime="2026-11-01 22:41:14.967484989 +0000 UTC m=+1078.751923911" Nov 01 22:41:15.089277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:15.089239 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9"] Nov 01 22:41:15.098716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:15.098680 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-mwkt9"] Nov 01 22:41:15.560132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:15.551944 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt"] Nov 01 22:41:15.561290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:15.560592 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerName="istio-proxy" containerID="cri-o://8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43" gracePeriod=30 Nov 01 22:41:15.639155 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:15.632938 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:15.700468 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:15.700434 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:15.821743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:15.821661 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:15.822539 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:15.822523 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:41:16.581382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:16.580897 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee812bbc-878f-4308-a1dc-e277443bcbcf" path="/var/lib/kubelet/pods/ee812bbc-878f-4308-a1dc-e277443bcbcf/volumes" Nov 01 22:41:17.276215 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:17.276167 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42"] Nov 01 22:41:17.276694 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:17.276407 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" podUID="5fe38402-a86d-4392-bea5-0452d9bebcae" containerName="maas-api" containerID="cri-o://8b3aaa8e6215fa751ca90208160e35d391091e095d47991c149347123f3b8216" gracePeriod=30 Nov 01 22:41:17.598210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:17.592632 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" podUID="5fe38402-a86d-4392-bea5-0452d9bebcae" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.113:8443/health\": dial tcp 10.134.0.113:8443: connect: connection refused" Nov 01 22:41:17.949133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:17.948436 6466 generic.go:356] "Generic (PLEG): container finished" podID="5fe38402-a86d-4392-bea5-0452d9bebcae" containerID="8b3aaa8e6215fa751ca90208160e35d391091e095d47991c149347123f3b8216" exitCode=0 Nov 01 22:41:17.950415 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:17.949607 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" event={"ID":"5fe38402-a86d-4392-bea5-0452d9bebcae","Type":"ContainerDied","Data":"8b3aaa8e6215fa751ca90208160e35d391091e095d47991c149347123f3b8216"} Nov 01 22:41:18.596017 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.595990 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:41:18.659070 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.659033 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-maas-api-tls\") pod \"5fe38402-a86d-4392-bea5-0452d9bebcae\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " Nov 01 22:41:18.659300 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.659249 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe38402-a86d-4392-bea5-0452d9bebcae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe38402-a86d-4392-bea5-0452d9bebcae-openshift-service-ca\") pod \"5fe38402-a86d-4392-bea5-0452d9bebcae\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " Nov 01 22:41:18.659381 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.659300 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe38402-a86d-4392-bea5-0452d9bebcae-kube-api-access-xhmtf\" (UniqueName: \"kubernetes.io/projected/5fe38402-a86d-4392-bea5-0452d9bebcae-kube-api-access-xhmtf\") pod \"5fe38402-a86d-4392-bea5-0452d9bebcae\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " Nov 01 22:41:18.659449 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.659402 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-metrics-certs\") pod \"5fe38402-a86d-4392-bea5-0452d9bebcae\" (UID: \"5fe38402-a86d-4392-bea5-0452d9bebcae\") " Nov 01 22:41:18.659984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.659954 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe38402-a86d-4392-bea5-0452d9bebcae-openshift-service-ca" pod "5fe38402-a86d-4392-bea5-0452d9bebcae" (UID: "5fe38402-a86d-4392-bea5-0452d9bebcae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:18.661212 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.661184 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe38402-a86d-4392-bea5-0452d9bebcae-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:18.661950 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.661922 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-metrics-certs" pod "5fe38402-a86d-4392-bea5-0452d9bebcae" (UID: "5fe38402-a86d-4392-bea5-0452d9bebcae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:18.662091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.661958 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-maas-api-tls" pod "5fe38402-a86d-4392-bea5-0452d9bebcae" (UID: "5fe38402-a86d-4392-bea5-0452d9bebcae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:18.662091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.661990 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe38402-a86d-4392-bea5-0452d9bebcae-kube-api-access-xhmtf" pod "5fe38402-a86d-4392-bea5-0452d9bebcae" (UID: "5fe38402-a86d-4392-bea5-0452d9bebcae"). InnerVolumeSpecName "kube-api-access-xhmtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:18.764417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.764329 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:18.764417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.764360 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhmtf\" (UniqueName: \"kubernetes.io/projected/5fe38402-a86d-4392-bea5-0452d9bebcae-kube-api-access-xhmtf\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:18.764417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.764373 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe38402-a86d-4392-bea5-0452d9bebcae-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:18.969177 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.969021 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" event={"ID":"5fe38402-a86d-4392-bea5-0452d9bebcae","Type":"ContainerDied","Data":"63242528a4936f1ca7a00579c3b878e6b6892bd9fd8d77834509f6c3fd025282"} Nov 01 22:41:18.969371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.969214 6466 scope.go:122] "RemoveContainer" containerID="8b3aaa8e6215fa751ca90208160e35d391091e095d47991c149347123f3b8216" Nov 01 22:41:18.969426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:18.969374 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42" Nov 01 22:41:19.004256 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.004197 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42"] Nov 01 22:41:19.013069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.013037 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-47x42"] Nov 01 22:41:19.108308 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.108221 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8"] Nov 01 22:41:19.109103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.109085 6466 cpu_manager.go:414] "containerMap: removing container" podUID="5fe38402-a86d-4392-bea5-0452d9bebcae" containerName="maas-api" Nov 01 22:41:19.109103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.109101 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe38402-a86d-4392-bea5-0452d9bebcae" containerName="maas-api" Nov 01 22:41:19.109284 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.109130 6466 cpu_manager.go:414] "containerMap: removing container" podUID="ee812bbc-878f-4308-a1dc-e277443bcbcf" containerName="payload-processing" Nov 01 22:41:19.109284 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.109136 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee812bbc-878f-4308-a1dc-e277443bcbcf" containerName="payload-processing" Nov 01 22:41:19.109455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.109344 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe38402-a86d-4392-bea5-0452d9bebcae" containerName="maas-api" Nov 01 22:41:19.109455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.109360 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee812bbc-878f-4308-a1dc-e277443bcbcf" containerName="payload-processing" Nov 01 22:41:19.114737 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.114716 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:41:19.125261 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.124630 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8"] Nov 01 22:41:19.125261 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.125255 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b4f6bb\"" Nov 01 22:41:19.146318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.146289 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5"] Nov 01 22:41:19.151334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.151311 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:41:19.177658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.177627 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5"] Nov 01 22:41:19.177890 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.177818 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b4f6bb-dockercfg-rbk2r\"" Nov 01 22:41:19.505132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.498031 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8\" (UID: \"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:41:19.505132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.498725 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwltl\" (UniqueName: \"kubernetes.io/projected/68f2b87e-7e90-47d7-b323-26bbf215051b-kube-api-access-jwltl\") pod \"payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5\" (UID: \"68f2b87e-7e90-47d7-b323-26bbf215051b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:41:19.505132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.498788 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f2b87e-7e90-47d7-b323-26bbf215051b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5\" (UID: \"68f2b87e-7e90-47d7-b323-26bbf215051b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:41:19.505132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.498907 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqpq5\" (UniqueName: \"kubernetes.io/projected/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-kube-api-access-tqpq5\") pod \"payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8\" (UID: \"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:41:19.611134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.606701 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwltl\" (UniqueName: \"kubernetes.io/projected/68f2b87e-7e90-47d7-b323-26bbf215051b-kube-api-access-jwltl\") pod \"payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5\" (UID: \"68f2b87e-7e90-47d7-b323-26bbf215051b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:41:19.611134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.606776 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f2b87e-7e90-47d7-b323-26bbf215051b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5\" (UID: \"68f2b87e-7e90-47d7-b323-26bbf215051b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:41:19.611134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.606879 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqpq5\" (UniqueName: \"kubernetes.io/projected/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-kube-api-access-tqpq5\") pod \"payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8\" (UID: \"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:41:19.611134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.606952 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8\" (UID: \"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:41:19.619191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.612981 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f2b87e-7e90-47d7-b323-26bbf215051b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5\" (UID: \"68f2b87e-7e90-47d7-b323-26bbf215051b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:41:19.627936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.627858 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8\" (UID: \"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:41:19.844196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:19.840020 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75"] Nov 01 22:41:20.132417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.132380 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwltl\" (UniqueName: \"kubernetes.io/projected/68f2b87e-7e90-47d7-b323-26bbf215051b-kube-api-access-jwltl\") pod \"payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5\" (UID: \"68f2b87e-7e90-47d7-b323-26bbf215051b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:41:20.391275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.167642 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqpq5\" (UniqueName: \"kubernetes.io/projected/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-kube-api-access-tqpq5\") pod \"payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8\" (UID: \"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:41:20.391275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.311930 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75"] Nov 01 22:41:20.391275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.312034 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.391275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.382165 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:41:20.396631 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.396548 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3e228e-openshift-default-dockercfg-jmr4n\"" Nov 01 22:41:20.407860 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.407826 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:41:20.508769 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.508732 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-token\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.511463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.511430 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjm7r\" (UniqueName: \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-kube-api-access-mjm7r\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.511694 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.511675 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istiod-ca-cert\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.511820 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.511805 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-podinfo\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.516164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.514361 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-envoy\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.516164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.514424 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-data\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.516164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.514544 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-credential-socket\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.516164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.514593 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-certs\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.516164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.514690 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-socket\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.561798 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.561754 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe38402-a86d-4392-bea5-0452d9bebcae" path="/var/lib/kubelet/pods/5fe38402-a86d-4392-bea5-0452d9bebcae/volumes" Nov 01 22:41:20.585857 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:41:20.585816 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:41:20.683869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.626087 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm7r\" (UniqueName: \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-kube-api-access-mjm7r\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.683869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.683628 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istiod-ca-cert\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.683869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.683661 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-podinfo\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.683869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.683750 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-envoy\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.683869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.683780 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-data\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.683869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.683849 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-credential-socket\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.684437 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.683883 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-certs\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.684437 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.683937 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-socket\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.684437 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.684099 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-token\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.685317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.685228 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istiod-ca-cert\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.685969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.685940 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-credential-socket\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.686054 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.686009 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-certs\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.686868 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.686646 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-socket\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.686965 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.686946 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-data\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.689650 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.689619 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-podinfo\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.689870 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.689836 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-envoy\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.690596 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.690568 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjm7r\" (UniqueName: \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-kube-api-access-mjm7r\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.699640 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.698854 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-token\") pod \"e2e-api-a-3e228e-openshift-default-5675fb8795-52d75\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:20.783347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:20.783265 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:21.276220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.275818 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:41:21.326855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.326346 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8"] Nov 01 22:41:21.326855 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:41:21.326356 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8221bb_bcff_4e2a_af5c_76f36b0a6477.slice/crio-c38bc7c86f4987cc633b4c146d4c63dacb18a0a327a9aeed898e7afea4fd9e90 WatchSource:0}: Error finding container c38bc7c86f4987cc633b4c146d4c63dacb18a0a327a9aeed898e7afea4fd9e90: Status 404 returned error can't find the container with id c38bc7c86f4987cc633b4c146d4c63dacb18a0a327a9aeed898e7afea4fd9e90 Nov 01 22:41:21.330554 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.330520 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:41:21.330664 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.330600 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:41:21.330724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.330676 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:41:21.333207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.333178 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75"] Nov 01 22:41:21.379175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.379147 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5"] Nov 01 22:41:21.380378 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:41:21.380352 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f2b87e_7e90_47d7_b323_26bbf215051b.slice/crio-689dc429a6ef07b7fb45f58c2e631ef75717fe86f0366661449285c69e059bfe WatchSource:0}: Error finding container 689dc429a6ef07b7fb45f58c2e631ef75717fe86f0366661449285c69e059bfe: Status 404 returned error can't find the container with id 689dc429a6ef07b7fb45f58c2e631ef75717fe86f0366661449285c69e059bfe Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.414308 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-credential-socket\") pod \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.414461 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-token\" (UniqueName: \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-token\") pod \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.414505 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-data\") pod \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.414558 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-kube-api-access-fddt6\" (UniqueName: \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-kube-api-access-fddt6\") pod \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.414627 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-credential-socket" pod "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" (UID: "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.414638 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-socket\") pod \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.414796 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-certs\") pod \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.414853 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istiod-ca-cert\") pod \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.415048 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-podinfo\") pod \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.415186 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-envoy\") pod \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\" (UID: \"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db\") " Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.416120 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.414853 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-data" pod "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" (UID: "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.416957 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-socket" pod "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" (UID: "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.416999 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-certs" pod "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" (UID: "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:21.417364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.417336 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istiod-ca-cert" pod "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" (UID: "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:21.424589 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.424535 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-token" pod "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" (UID: "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:21.455677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.455480 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-envoy" pod "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" (UID: "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:21.455677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.455629 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-kube-api-access-fddt6" pod "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" (UID: "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db"). InnerVolumeSpecName "kube-api-access-fddt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:21.462811 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.462722 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-podinfo" pod "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" (UID: "3c3f43b1-c00b-4e77-87cd-cba74f4ba1db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:41:21.518223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.518152 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:21.518223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.518191 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:21.518796 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.518678 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:21.518967 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.518727 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:21.519223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.519091 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:21.519223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.519137 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:21.519223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.519156 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fddt6\" (UniqueName: \"kubernetes.io/projected/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-kube-api-access-fddt6\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:21.519223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.519172 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:21.807989 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.807951 6466 generic.go:356] "Generic (PLEG): container finished" podID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerID="8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43" exitCode=0 Nov 01 22:41:21.825132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.808658 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" event={"ID":"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db","Type":"ContainerDied","Data":"8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43"} Nov 01 22:41:21.825132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.808732 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" event={"ID":"3c3f43b1-c00b-4e77-87cd-cba74f4ba1db","Type":"ContainerDied","Data":"fc3964d23b6ac0328c9241a24d772e17962ef4ca7e76d584616ee081b9a55a85"} Nov 01 22:41:21.825132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.808764 6466 scope.go:122] "RemoveContainer" containerID="8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43" Nov 01 22:41:21.825132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.808908 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt" Nov 01 22:41:21.864305 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.864244 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" event={"ID":"68f2b87e-7e90-47d7-b323-26bbf215051b","Type":"ContainerStarted","Data":"44e758b8a7582ba22ffaf84bd02b819c7823d2bed979b5c266fc357de366d6bd"} Nov 01 22:41:21.868461 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.868427 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" event={"ID":"68f2b87e-7e90-47d7-b323-26bbf215051b","Type":"ContainerStarted","Data":"689dc429a6ef07b7fb45f58c2e631ef75717fe86f0366661449285c69e059bfe"} Nov 01 22:41:21.878302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.877532 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:21.878302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.877600 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.878302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.877928 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:41:21.880962 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.880928 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" event={"ID":"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6","Type":"ContainerStarted","Data":"64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff"} Nov 01 22:41:21.881135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.880996 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" event={"ID":"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6","Type":"ContainerStarted","Data":"916f9c9061640db8984f6ad00c7c7835b39e466dc05112cfe628ad975a5c16c5"} Nov 01 22:41:21.882072 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.882044 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:41:21.884960 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.884933 6466 scope.go:122] "RemoveContainer" containerID="8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43" Nov 01 22:41:21.890435 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:41:21.889566 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43\": container with ID starting with 8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43 not found: ID does not exist" containerID="8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43" Nov 01 22:41:21.890435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.889603 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43"} err="failed to get container status \"8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43\": rpc error: code = NotFound desc = could not find container \"8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43\": container with ID starting with 8bda767f6344fb3d6b8eb8cf6424c81c316dbffeaa5319cd15da3f4a7988bc43 not found: ID does not exist" Nov 01 22:41:21.891803 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.891769 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" event={"ID":"2f8221bb-bcff-4e2a-af5c-76f36b0a6477","Type":"ContainerStarted","Data":"a661736aecfbf2f3003ca2e286777d9b4362dfe43e99d7ef2dcad773b161b487"} Nov 01 22:41:21.891912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.891836 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" event={"ID":"2f8221bb-bcff-4e2a-af5c-76f36b0a6477","Type":"ContainerStarted","Data":"c38bc7c86f4987cc633b4c146d4c63dacb18a0a327a9aeed898e7afea4fd9e90"} Nov 01 22:41:21.904888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.904834 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" podStartSLOduration=2.904817813 podStartE2EDuration="2.904817813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:21.89803266 +0000 UTC m=+1085.682471582" watchObservedRunningTime="2026-11-01 22:41:21.904817813 +0000 UTC m=+1085.689256734" Nov 01 22:41:21.920251 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.919207 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt"] Nov 01 22:41:21.927123 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.927080 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6vmrt"] Nov 01 22:41:21.941753 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.939101 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" podStartSLOduration=2.939082526 podStartE2EDuration="2.939082526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:21.93544573 +0000 UTC m=+1085.719884643" watchObservedRunningTime="2026-11-01 22:41:21.939082526 +0000 UTC m=+1085.723521447" Nov 01 22:41:21.964085 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:21.964028 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podStartSLOduration=2.96401294 podStartE2EDuration="2.96401294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:21.959493002 +0000 UTC m=+1085.743931923" watchObservedRunningTime="2026-11-01 22:41:21.96401294 +0000 UTC m=+1085.748451861" Nov 01 22:41:22.345485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:22.345451 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" path="/var/lib/kubelet/pods/3c3f43b1-c00b-4e77-87cd-cba74f4ba1db/volumes" Nov 01 22:41:22.784140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:22.784070 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:22.791184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:22.791156 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:22.903239 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:22.903211 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:22.904422 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:22.904400 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:41:24.922177 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:24.922142 6466 generic.go:356] "Generic (PLEG): container finished" podID="101175a2-a477-494a-99ff-ac688c53c07e" containerID="b01b22ec0d0d3a6a351b46ad2b60a5f7a70ee174fe1cbe3fcd146eca65e4e4d6" exitCode=0 Nov 01 22:41:24.922177 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:24.922157 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" event={"ID":"101175a2-a477-494a-99ff-ac688c53c07e","Type":"ContainerDied","Data":"b01b22ec0d0d3a6a351b46ad2b60a5f7a70ee174fe1cbe3fcd146eca65e4e4d6"} Nov 01 22:41:24.922925 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:24.922908 6466 scope.go:122] "RemoveContainer" containerID="b01b22ec0d0d3a6a351b46ad2b60a5f7a70ee174fe1cbe3fcd146eca65e4e4d6" Nov 01 22:41:24.924647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:24.924621 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:41:24.924731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:24.924694 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:41:24.924731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:24.924727 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:41:25.941135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:25.941041 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" event={"ID":"101175a2-a477-494a-99ff-ac688c53c07e","Type":"ContainerStarted","Data":"e8db45fdaf8a5ebe4aa15781d4630f8521596de384e9b1bac70388560788f059"} Nov 01 22:41:26.174239 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.174159 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6"] Nov 01 22:41:26.175260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.175240 6466 cpu_manager.go:414] "containerMap: removing container" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerName="istio-proxy" Nov 01 22:41:26.175260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.175259 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerName="istio-proxy" Nov 01 22:41:26.175423 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.175414 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3f43b1-c00b-4e77-87cd-cba74f4ba1db" containerName="istio-proxy" Nov 01 22:41:26.179501 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.179485 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:41:26.191525 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.191470 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-425870c1\"" Nov 01 22:41:26.194120 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.191758 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6"] Nov 01 22:41:26.213271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.213239 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct"] Nov 01 22:41:26.217719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.217254 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-425870c1-dockercfg-k5j4s\"" Nov 01 22:41:26.502363 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.481782 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k57t\" (UniqueName: \"kubernetes.io/projected/707f0b16-452d-4adc-ac7d-5e3cb927618f-kube-api-access-8k57t\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6\" (UID: \"707f0b16-452d-4adc-ac7d-5e3cb927618f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:41:26.502363 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.482342 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707f0b16-452d-4adc-ac7d-5e3cb927618f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6\" (UID: \"707f0b16-452d-4adc-ac7d-5e3cb927618f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:41:26.765358 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.765272 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k57t\" (UniqueName: \"kubernetes.io/projected/707f0b16-452d-4adc-ac7d-5e3cb927618f-kube-api-access-8k57t\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6\" (UID: \"707f0b16-452d-4adc-ac7d-5e3cb927618f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:41:26.766603 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.766568 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhchq\" (UniqueName: \"kubernetes.io/projected/49592222-eba1-4751-98e0-0aaba69e8167-kube-api-access-zhchq\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct\" (UID: \"49592222-eba1-4751-98e0-0aaba69e8167\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:41:26.766983 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.766961 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707f0b16-452d-4adc-ac7d-5e3cb927618f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6\" (UID: \"707f0b16-452d-4adc-ac7d-5e3cb927618f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:41:26.768167 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.768146 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49592222-eba1-4751-98e0-0aaba69e8167-plugins-config-volume\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct\" (UID: \"49592222-eba1-4751-98e0-0aaba69e8167\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:41:26.768617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.767972 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707f0b16-452d-4adc-ac7d-5e3cb927618f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6\" (UID: \"707f0b16-452d-4adc-ac7d-5e3cb927618f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:41:26.870133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.846140 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 22:41:26.870133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.846230 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 22:41:26.870133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.847267 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:41:26.941335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.941299 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhchq\" (UniqueName: \"kubernetes.io/projected/49592222-eba1-4751-98e0-0aaba69e8167-kube-api-access-zhchq\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct\" (UID: \"49592222-eba1-4751-98e0-0aaba69e8167\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:41:26.955897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.955866 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49592222-eba1-4751-98e0-0aaba69e8167-plugins-config-volume\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct\" (UID: \"49592222-eba1-4751-98e0-0aaba69e8167\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:41:26.957674 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:26.957645 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49592222-eba1-4751-98e0-0aaba69e8167-plugins-config-volume\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct\" (UID: \"49592222-eba1-4751-98e0-0aaba69e8167\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:41:27.057323 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.044895 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k57t\" (UniqueName: \"kubernetes.io/projected/707f0b16-452d-4adc-ac7d-5e3cb927618f-kube-api-access-8k57t\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6\" (UID: \"707f0b16-452d-4adc-ac7d-5e3cb927618f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:41:27.110134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.086999 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhchq\" (UniqueName: \"kubernetes.io/projected/49592222-eba1-4751-98e0-0aaba69e8167-kube-api-access-zhchq\") pod \"payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct\" (UID: \"49592222-eba1-4751-98e0-0aaba69e8167\") " pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:41:27.147581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.124752 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:41:27.147581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.131437 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:41:27.614004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.606978 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:41:27.614004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.607087 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:41:27.614004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.607252 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct"] Nov 01 22:41:27.614004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.607293 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47"] Nov 01 22:41:27.737131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.736609 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47"] Nov 01 22:41:27.737131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.736661 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp"] Nov 01 22:41:27.760869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.738576 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 22:41:27.760869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.738724 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 22:41:27.777276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.777246 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:41:27.785642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.785610 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:41:27.796153 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.790832 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3e228e\"" Nov 01 22:41:27.796153 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.791592 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3e228e-dockercfg-tnrwj\"" Nov 01 22:41:27.847672 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.847620 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp"] Nov 01 22:41:27.903476 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.903440 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4pd\" (UniqueName: \"kubernetes.io/projected/de632f88-68c6-4e45-bef6-301d8350dd60-kube-api-access-6c4pd\") pod \"payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47\" (UID: \"de632f88-68c6-4e45-bef6-301d8350dd60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:41:27.914968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:27.904328 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de632f88-68c6-4e45-bef6-301d8350dd60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47\" (UID: \"de632f88-68c6-4e45-bef6-301d8350dd60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:41:28.034389 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.026943 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de632f88-68c6-4e45-bef6-301d8350dd60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47\" (UID: \"de632f88-68c6-4e45-bef6-301d8350dd60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:41:28.034389 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.034312 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de632f88-68c6-4e45-bef6-301d8350dd60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47\" (UID: \"de632f88-68c6-4e45-bef6-301d8350dd60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:41:28.116180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.115279 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e228e-98b4865b9-88klp\" (UID: \"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:41:28.116180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.115563 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qd6l\" (UniqueName: \"kubernetes.io/projected/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-kube-api-access-8qd6l\") pod \"payload-processing-e2e-api-a-3e228e-98b4865b9-88klp\" (UID: \"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:41:28.116180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.115675 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4pd\" (UniqueName: \"kubernetes.io/projected/de632f88-68c6-4e45-bef6-301d8350dd60-kube-api-access-6c4pd\") pod \"payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47\" (UID: \"de632f88-68c6-4e45-bef6-301d8350dd60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:41:28.140355 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.140322 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4pd\" (UniqueName: \"kubernetes.io/projected/de632f88-68c6-4e45-bef6-301d8350dd60-kube-api-access-6c4pd\") pod \"payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47\" (UID: \"de632f88-68c6-4e45-bef6-301d8350dd60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:41:28.230164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.218302 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e228e-98b4865b9-88klp\" (UID: \"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:41:28.230164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.218546 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qd6l\" (UniqueName: \"kubernetes.io/projected/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-kube-api-access-8qd6l\") pod \"payload-processing-e2e-api-a-3e228e-98b4865b9-88klp\" (UID: \"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:41:28.230164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.224215 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3e228e-98b4865b9-88klp\" (UID: \"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:41:28.277617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.245946 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qd6l\" (UniqueName: \"kubernetes.io/projected/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-kube-api-access-8qd6l\") pod \"payload-processing-e2e-api-a-3e228e-98b4865b9-88klp\" (UID: \"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a\") " pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:41:28.277617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.255465 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:41:28.319060 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.319015 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:41:28.771006 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.770970 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 22:41:28.771291 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.771261 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 22:41:28.926803 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:28.911793 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6"] Nov 01 22:41:29.223285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:29.223250 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" event={"ID":"707f0b16-452d-4adc-ac7d-5e3cb927618f","Type":"ContainerStarted","Data":"ca7f7b62465aa77a1654fdae3214ef379321eeb71aedb5f4b5aaf83e47581ccd"} Nov 01 22:41:29.223740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:29.223306 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" event={"ID":"707f0b16-452d-4adc-ac7d-5e3cb927618f","Type":"ContainerStarted","Data":"029820a739159abd7da003b6fc89c0f743e91fba363dcd3001fbd867c10a0eb6"} Nov 01 22:41:29.223740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:29.223338 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:41:29.276371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:29.276257 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" podStartSLOduration=3.276236466 podStartE2EDuration="3.276236466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:29.24580188 +0000 UTC m=+1093.030240801" watchObservedRunningTime="2026-11-01 22:41:29.276236466 +0000 UTC m=+1093.060675388" Nov 01 22:41:29.299048 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:41:29.299020 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde632f88_68c6_4e45_bef6_301d8350dd60.slice/crio-fc8d1c12fd059bf8492da4ae4a595d883427ac3b6951bcfd3cba234e15e7b611 WatchSource:0}: Error finding container fc8d1c12fd059bf8492da4ae4a595d883427ac3b6951bcfd3cba234e15e7b611: Status 404 returned error can't find the container with id fc8d1c12fd059bf8492da4ae4a595d883427ac3b6951bcfd3cba234e15e7b611 Nov 01 22:41:29.300219 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:29.300196 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47"] Nov 01 22:41:29.319772 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:29.319748 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp"] Nov 01 22:41:29.320198 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:41:29.320097 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516adbc4_8ec9_42a2_8c3b_868e5bd73c9a.slice/crio-faf66a17671904f4b796de6ff5e3483b3234b66a82a495b0bf14441aab493143 WatchSource:0}: Error finding container faf66a17671904f4b796de6ff5e3483b3234b66a82a495b0bf14441aab493143: Status 404 returned error can't find the container with id faf66a17671904f4b796de6ff5e3483b3234b66a82a495b0bf14441aab493143 Nov 01 22:41:29.439135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:29.421594 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct"] Nov 01 22:41:29.698170 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:29.696738 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 22:41:29.698170 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:29.696833 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 22:41:30.235065 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.235028 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" event={"ID":"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a","Type":"ContainerStarted","Data":"6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73"} Nov 01 22:41:30.245688 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.235094 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" event={"ID":"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a","Type":"ContainerStarted","Data":"faf66a17671904f4b796de6ff5e3483b3234b66a82a495b0bf14441aab493143"} Nov 01 22:41:30.245688 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.235147 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:41:30.271447 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.252413 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" event={"ID":"de632f88-68c6-4e45-bef6-301d8350dd60","Type":"ContainerStarted","Data":"21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915"} Nov 01 22:41:30.271447 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.269498 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:41:30.271447 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.269541 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" event={"ID":"de632f88-68c6-4e45-bef6-301d8350dd60","Type":"ContainerStarted","Data":"fc8d1c12fd059bf8492da4ae4a595d883427ac3b6951bcfd3cba234e15e7b611"} Nov 01 22:41:30.273353 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.273321 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" event={"ID":"49592222-eba1-4751-98e0-0aaba69e8167","Type":"ContainerStarted","Data":"7ccdc531ecaede15c2eec9d80ce44a316e5c99b9f6324524db82062cf164783e"} Nov 01 22:41:30.273640 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.273382 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" event={"ID":"49592222-eba1-4751-98e0-0aaba69e8167","Type":"ContainerStarted","Data":"fd6627583e45332a41b7e73e194a9d2167c2174c0f7e6b7c308d7b65738a1972"} Nov 01 22:41:30.273640 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.273409 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" podStartSLOduration=3.273394108 podStartE2EDuration="3.273394108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:30.269623326 +0000 UTC m=+1094.054062248" watchObservedRunningTime="2026-11-01 22:41:30.273394108 +0000 UTC m=+1094.057833030" Nov 01 22:41:30.273640 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.273562 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:41:30.296703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.296639 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" podStartSLOduration=3.296623671 podStartE2EDuration="3.296623671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:30.292419477 +0000 UTC m=+1094.076858403" watchObservedRunningTime="2026-11-01 22:41:30.296623671 +0000 UTC m=+1094.081062592" Nov 01 22:41:30.697538 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.697496 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 22:41:30.697845 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:30.697813 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 22:41:31.694414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:31.694378 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 22:41:31.694789 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:31.694436 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 22:41:32.702135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:32.695408 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 01 22:41:32.702135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:32.695470 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 01 22:41:32.906064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:32.906032 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:41:32.931734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:32.931673 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" podStartSLOduration=6.931652866 podStartE2EDuration="6.931652866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:30.315967305 +0000 UTC m=+1094.100406226" watchObservedRunningTime="2026-11-01 22:41:32.931652866 +0000 UTC m=+1096.716091788" Nov 01 22:41:33.699232 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:33.699203 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:41:33.699739 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:33.699721 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:41:33.700643 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:33.700627 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:41:33.916067 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:33.916040 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:41:40.276784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:40.276757 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:41:41.284903 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:41.284870 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:41:41.285367 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:41.284943 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:41:41.285367 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:41:41.285183 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:41:49.504135 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:41:49.499024 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:42:00.879056 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:00.878515 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr"] Nov 01 22:42:00.879056 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:00.878777 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" containerID="cri-o://aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c" gracePeriod=30 Nov 01 22:42:06.187636 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.187610 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:42:06.286306 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.286272 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5"] Nov 01 22:42:06.287300 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.287278 6466 cpu_manager.go:414] "containerMap: removing container" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" Nov 01 22:42:06.287300 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.287300 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" Nov 01 22:42:06.287499 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.287485 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerName="istio-proxy" Nov 01 22:42:06.291384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.291361 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.312750 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.312721 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9b7320-openshift-default-dockercfg-zmkgf\"" Nov 01 22:42:06.333019 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.332988 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5"] Nov 01 22:42:06.356459 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.356424 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-podinfo\") pod \"3b5c15c5-4f15-43d2-b26c-2ea276334184\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " Nov 01 22:42:06.356651 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.356521 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5c15c5-4f15-43d2-b26c-2ea276334184-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5c15c5-4f15-43d2-b26c-2ea276334184-istiod-ca-cert\") pod \"3b5c15c5-4f15-43d2-b26c-2ea276334184\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " Nov 01 22:42:06.356651 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.356575 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-kube-api-access-wkfdj\" (UniqueName: \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-kube-api-access-wkfdj\") pod \"3b5c15c5-4f15-43d2-b26c-2ea276334184\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " Nov 01 22:42:06.356813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.356723 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-credential-socket\") pod \"3b5c15c5-4f15-43d2-b26c-2ea276334184\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " Nov 01 22:42:06.357291 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.356978 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-envoy\") pod \"3b5c15c5-4f15-43d2-b26c-2ea276334184\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " Nov 01 22:42:06.357541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.357502 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-credential-socket" pod "3b5c15c5-4f15-43d2-b26c-2ea276334184" (UID: "3b5c15c5-4f15-43d2-b26c-2ea276334184"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:06.357718 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.357684 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-data\") pod \"3b5c15c5-4f15-43d2-b26c-2ea276334184\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " Nov 01 22:42:06.357817 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.357790 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-certs\") pod \"3b5c15c5-4f15-43d2-b26c-2ea276334184\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " Nov 01 22:42:06.357899 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.357831 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-socket\") pod \"3b5c15c5-4f15-43d2-b26c-2ea276334184\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " Nov 01 22:42:06.357899 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.357844 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5c15c5-4f15-43d2-b26c-2ea276334184-istiod-ca-cert" pod "3b5c15c5-4f15-43d2-b26c-2ea276334184" (UID: "3b5c15c5-4f15-43d2-b26c-2ea276334184"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:06.358081 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.357982 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-token\" (UniqueName: \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-token\") pod \"3b5c15c5-4f15-43d2-b26c-2ea276334184\" (UID: \"3b5c15c5-4f15-43d2-b26c-2ea276334184\") " Nov 01 22:42:06.358178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.358093 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-certs" pod "3b5c15c5-4f15-43d2-b26c-2ea276334184" (UID: "3b5c15c5-4f15-43d2-b26c-2ea276334184"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:06.358529 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.358481 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-socket" pod "3b5c15c5-4f15-43d2-b26c-2ea276334184" (UID: "3b5c15c5-4f15-43d2-b26c-2ea276334184"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:06.358529 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.358498 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-data" pod "3b5c15c5-4f15-43d2-b26c-2ea276334184" (UID: "3b5c15c5-4f15-43d2-b26c-2ea276334184"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:06.359361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.359338 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5c15c5-4f15-43d2-b26c-2ea276334184-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:06.359462 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.359367 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:06.359462 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.359384 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:06.359462 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.359416 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:06.359462 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.359432 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:06.359862 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.359837 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-podinfo" pod "3b5c15c5-4f15-43d2-b26c-2ea276334184" (UID: "3b5c15c5-4f15-43d2-b26c-2ea276334184"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:42:06.360513 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.360491 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-envoy" pod "3b5c15c5-4f15-43d2-b26c-2ea276334184" (UID: "3b5c15c5-4f15-43d2-b26c-2ea276334184"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:06.361758 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.361731 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-kube-api-access-wkfdj" pod "3b5c15c5-4f15-43d2-b26c-2ea276334184" (UID: "3b5c15c5-4f15-43d2-b26c-2ea276334184"). InnerVolumeSpecName "kube-api-access-wkfdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:06.361870 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.361849 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-token" pod "3b5c15c5-4f15-43d2-b26c-2ea276334184" (UID: "3b5c15c5-4f15-43d2-b26c-2ea276334184"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:06.460964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.460874 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-envoy\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.461157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.460942 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-token\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.461237 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.461157 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-certs\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.461237 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.461197 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-podinfo\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.461362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.461314 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41e060e-e33f-478c-8019-ede84aeb4d13-istiod-ca-cert\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.461493 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.461478 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-socket\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.461560 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.461508 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-credential-socket\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.461560 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.461536 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-data\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.461678 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.461656 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7n7f\" (UniqueName: \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-kube-api-access-b7n7f\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.461872 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.461818 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:06.461913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.461877 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:06.461913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.461888 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkfdj\" (UniqueName: \"kubernetes.io/projected/3b5c15c5-4f15-43d2-b26c-2ea276334184-kube-api-access-wkfdj\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:06.461913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.461899 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5c15c5-4f15-43d2-b26c-2ea276334184-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:06.523956 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.523921 6466 generic.go:356] "Generic (PLEG): container finished" podID="3b5c15c5-4f15-43d2-b26c-2ea276334184" containerID="aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c" exitCode=0 Nov 01 22:42:06.524199 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.524011 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" Nov 01 22:42:06.524199 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.524010 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" event={"ID":"3b5c15c5-4f15-43d2-b26c-2ea276334184","Type":"ContainerDied","Data":"aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c"} Nov 01 22:42:06.524199 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.524103 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr" event={"ID":"3b5c15c5-4f15-43d2-b26c-2ea276334184","Type":"ContainerDied","Data":"c6d58eee70ac2019405040c5a4ebb53113e1ec852236df44703f1affc52435e4"} Nov 01 22:42:06.524199 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.524148 6466 scope.go:122] "RemoveContainer" containerID="aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c" Nov 01 22:42:06.537637 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.537616 6466 scope.go:122] "RemoveContainer" containerID="aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c" Nov 01 22:42:06.537913 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:42:06.537894 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c\": container with ID starting with aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c not found: ID does not exist" containerID="aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c" Nov 01 22:42:06.537955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.537933 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c"} err="failed to get container status \"aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c\": rpc error: code = NotFound desc = could not find container \"aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c\": container with ID starting with aacc1c023f4988d2ff2caeb4a535dc175714f21d1ef0c0f9413b217f99c2f11c not found: ID does not exist" Nov 01 22:42:06.553632 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.553606 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr"] Nov 01 22:42:06.563718 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.563685 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-credential-socket\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.563832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.563728 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-data\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.563832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.563775 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7n7f\" (UniqueName: \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-kube-api-access-b7n7f\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.563968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.563869 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-envoy\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.563968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.563936 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-token\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.564081 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.564008 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-credential-socket\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.564165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.564088 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-certs\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.564165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.564130 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-data\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.564165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.564133 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-podinfo\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.564317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.564259 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41e060e-e33f-478c-8019-ede84aeb4d13-istiod-ca-cert\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.564414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.564395 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-certs\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.564460 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.564443 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-socket\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.564755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.564738 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-socket\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.564893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.564872 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41e060e-e33f-478c-8019-ede84aeb4d13-istiod-ca-cert\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.566487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.566467 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-podinfo\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.566567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.566523 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-envoy\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.566875 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.566858 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-jqtgr"] Nov 01 22:42:06.574706 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.574686 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7n7f\" (UniqueName: \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-kube-api-access-b7n7f\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.575231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.575214 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-token\") pod \"e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.607452 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.607416 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:06.981174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.981071 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5"] Nov 01 22:42:06.983680 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.983646 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:06.983778 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.983717 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:06.983778 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:06.983748 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:07.537078 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:07.537034 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" event={"ID":"f41e060e-e33f-478c-8019-ede84aeb4d13","Type":"ContainerStarted","Data":"64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e"} Nov 01 22:42:07.537580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:07.537148 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" event={"ID":"f41e060e-e33f-478c-8019-ede84aeb4d13","Type":"ContainerStarted","Data":"9f9b7934a41dc46c2b2f9464c6d3d27456acad3657a55e50f2ad09f8095ce827"} Nov 01 22:42:07.604247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:07.604200 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" podStartSLOduration=18.604185682 podStartE2EDuration="18.604185682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:07.600314224 +0000 UTC m=+1131.384753239" watchObservedRunningTime="2026-11-01 22:42:07.604185682 +0000 UTC m=+1131.388624603" Nov 01 22:42:07.607522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:07.607498 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:08.414255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:08.414220 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5c15c5-4f15-43d2-b26c-2ea276334184" path="/var/lib/kubelet/pods/3b5c15c5-4f15-43d2-b26c-2ea276334184/volumes" Nov 01 22:42:08.614279 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:08.614251 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:09.556354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:09.556323 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:09.557359 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:09.557336 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:42:11.872911 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:11.872875 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk"] Nov 01 22:42:11.878833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:11.878810 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:42:11.908687 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:11.908642 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk"] Nov 01 22:42:11.909799 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:11.909775 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9b7320\"" Nov 01 22:42:11.910037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:11.910006 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9b7320-dockercfg-vkq5p\"" Nov 01 22:42:12.229140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:12.225878 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c6daab-48e9-4de4-b639-90f212867977-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk\" (UID: \"70c6daab-48e9-4de4-b639-90f212867977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:42:12.421226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:12.415861 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljnj4\" (UniqueName: \"kubernetes.io/projected/70c6daab-48e9-4de4-b639-90f212867977-kube-api-access-ljnj4\") pod \"payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk\" (UID: \"70c6daab-48e9-4de4-b639-90f212867977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:42:12.517435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:12.517350 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c6daab-48e9-4de4-b639-90f212867977-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk\" (UID: \"70c6daab-48e9-4de4-b639-90f212867977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:42:12.517603 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:12.517590 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljnj4\" (UniqueName: \"kubernetes.io/projected/70c6daab-48e9-4de4-b639-90f212867977-kube-api-access-ljnj4\") pod \"payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk\" (UID: \"70c6daab-48e9-4de4-b639-90f212867977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:42:12.518052 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:12.518029 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c6daab-48e9-4de4-b639-90f212867977-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk\" (UID: \"70c6daab-48e9-4de4-b639-90f212867977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:42:12.526662 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:12.526636 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljnj4\" (UniqueName: \"kubernetes.io/projected/70c6daab-48e9-4de4-b639-90f212867977-kube-api-access-ljnj4\") pod \"payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk\" (UID: \"70c6daab-48e9-4de4-b639-90f212867977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:42:12.803227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:12.795444 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:42:13.121046 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:42:13.121009 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c6daab_48e9_4de4_b639_90f212867977.slice/crio-5319c8471ea6d279565a254a350151cb40156ff9597b50b120a8e2cfb30d6290 WatchSource:0}: Error finding container 5319c8471ea6d279565a254a350151cb40156ff9597b50b120a8e2cfb30d6290: Status 404 returned error can't find the container with id 5319c8471ea6d279565a254a350151cb40156ff9597b50b120a8e2cfb30d6290 Nov 01 22:42:13.121497 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:13.121456 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk"] Nov 01 22:42:13.663549 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:13.663325 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" event={"ID":"70c6daab-48e9-4de4-b639-90f212867977","Type":"ContainerStarted","Data":"b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504"} Nov 01 22:42:13.663549 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:13.663398 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" event={"ID":"70c6daab-48e9-4de4-b639-90f212867977","Type":"ContainerStarted","Data":"5319c8471ea6d279565a254a350151cb40156ff9597b50b120a8e2cfb30d6290"} Nov 01 22:42:13.664617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:13.664580 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:42:13.723942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:13.723729 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" podStartSLOduration=2.723704488 podStartE2EDuration="2.723704488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:13.710336231 +0000 UTC m=+1137.494775153" watchObservedRunningTime="2026-11-01 22:42:13.723704488 +0000 UTC m=+1137.508143410" Nov 01 22:42:15.372278 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:15.372243 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv"] Nov 01 22:42:15.372716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:15.372455 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" containerID="cri-o://035d24c71c565cb8d1404240ff5377a54b982c344fa13bb21d62df809ddcfcbd" gracePeriod=30 Nov 01 22:42:15.672128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:15.671331 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4"] Nov 01 22:42:15.672128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:15.671584 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" podUID="04419d55-67d8-420d-8f48-f85969496c9d" containerName="istio-proxy" containerID="cri-o://0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50" gracePeriod=30 Nov 01 22:42:15.871340 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:15.862926 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z"] Nov 01 22:42:15.871340 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:15.863454 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" containerID="cri-o://512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55" gracePeriod=30 Nov 01 22:42:19.305653 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:42:19.305610 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:42:20.795838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.795805 6466 generic.go:356] "Generic (PLEG): container finished" podID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerID="035d24c71c565cb8d1404240ff5377a54b982c344fa13bb21d62df809ddcfcbd" exitCode=0 Nov 01 22:42:20.796260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.795877 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" event={"ID":"131d0e2f-d0ba-4623-8bdb-e7b969c526db","Type":"ContainerDied","Data":"035d24c71c565cb8d1404240ff5377a54b982c344fa13bb21d62df809ddcfcbd"} Nov 01 22:42:20.796260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.795946 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" event={"ID":"131d0e2f-d0ba-4623-8bdb-e7b969c526db","Type":"ContainerDied","Data":"36ac3f044b9909965d85b6ede61c50d85b00e036042a1b3874d073aab7464553"} Nov 01 22:42:20.796260 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.795977 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36ac3f044b9909965d85b6ede61c50d85b00e036042a1b3874d073aab7464553" Nov 01 22:42:20.825413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.825374 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:42:20.898156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.891348 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9"] Nov 01 22:42:20.947228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.944800 6466 cpu_manager.go:414] "containerMap: removing container" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" Nov 01 22:42:20.947228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.944831 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" Nov 01 22:42:20.947228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.946699 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-podinfo\") pod \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " Nov 01 22:42:20.947228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.946808 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-kube-api-access-mr42f\" (UniqueName: \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-kube-api-access-mr42f\") pod \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " Nov 01 22:42:20.947228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.946912 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-certs\") pod \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " Nov 01 22:42:20.947228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.946981 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-credential-socket\") pod \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " Nov 01 22:42:20.947228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.947057 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-socket\") pod \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " Nov 01 22:42:20.947578 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.947257 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" containerName="istio-proxy" Nov 01 22:42:20.947953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.947924 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-certs" pod "131d0e2f-d0ba-4623-8bdb-e7b969c526db" (UID: "131d0e2f-d0ba-4623-8bdb-e7b969c526db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:20.951276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.948238 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-envoy\") pod \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " Nov 01 22:42:20.951276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.948357 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istiod-ca-cert\") pod \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " Nov 01 22:42:20.951276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.948440 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-token\" (UniqueName: \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-token\") pod \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " Nov 01 22:42:20.951276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.948684 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-data\") pod \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\" (UID: \"131d0e2f-d0ba-4623-8bdb-e7b969c526db\") " Nov 01 22:42:20.963690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.963561 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-podinfo" pod "131d0e2f-d0ba-4623-8bdb-e7b969c526db" (UID: "131d0e2f-d0ba-4623-8bdb-e7b969c526db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:42:20.963827 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.963804 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-credential-socket" pod "131d0e2f-d0ba-4623-8bdb-e7b969c526db" (UID: "131d0e2f-d0ba-4623-8bdb-e7b969c526db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:20.981185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.964016 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-socket" pod "131d0e2f-d0ba-4623-8bdb-e7b969c526db" (UID: "131d0e2f-d0ba-4623-8bdb-e7b969c526db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:20.981185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.964401 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istiod-ca-cert" pod "131d0e2f-d0ba-4623-8bdb-e7b969c526db" (UID: "131d0e2f-d0ba-4623-8bdb-e7b969c526db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:20.981185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.969744 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-data" pod "131d0e2f-d0ba-4623-8bdb-e7b969c526db" (UID: "131d0e2f-d0ba-4623-8bdb-e7b969c526db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:20.996307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.996120 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-token" pod "131d0e2f-d0ba-4623-8bdb-e7b969c526db" (UID: "131d0e2f-d0ba-4623-8bdb-e7b969c526db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:20.997430 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:20.997349 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-kube-api-access-mr42f" pod "131d0e2f-d0ba-4623-8bdb-e7b969c526db" (UID: "131d0e2f-d0ba-4623-8bdb-e7b969c526db"). InnerVolumeSpecName "kube-api-access-mr42f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:21.008591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.007157 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.008591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.007191 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.008591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.007220 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.008591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.007236 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.008591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.007254 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.008591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.007270 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.012478 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.010432 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-envoy" pod "131d0e2f-d0ba-4623-8bdb-e7b969c526db" (UID: "131d0e2f-d0ba-4623-8bdb-e7b969c526db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:21.012478 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.011538 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9"] Nov 01 22:42:21.012478 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.011660 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.016635 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.016290 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-dcd5d4-openshift-default-dockercfg-5jlfp\"" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110083 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110275 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4z6x\" (UniqueName: \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-kube-api-access-t4z6x\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110323 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110372 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110435 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110541 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110696 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110744 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110783 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110888 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr42f\" (UniqueName: \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-kube-api-access-mr42f\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110906 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.116142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.110919 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131d0e2f-d0ba-4623-8bdb-e7b969c526db-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.215004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.213691 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.215004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.213757 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.215004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.213795 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.215004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.213946 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.215004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.214065 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4z6x\" (UniqueName: \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-kube-api-access-t4z6x\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.215004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.214179 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.215004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.214234 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.215004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.214297 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.215004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.214406 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.215618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.215134 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.215618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.215255 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.215618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.215389 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.216006 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.215977 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.216248 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.216228 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.216424 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.216400 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:42:21.235898 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.232506 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.243140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.238831 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.276282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.245129 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4z6x\" (UniqueName: \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-kube-api-access-t4z6x\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.276282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.250037 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.333324 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.333287 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg"] Nov 01 22:42:21.334984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.334955 6466 cpu_manager.go:414] "containerMap: removing container" podUID="04419d55-67d8-420d-8f48-f85969496c9d" containerName="istio-proxy" Nov 01 22:42:21.335187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.335168 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04419d55-67d8-420d-8f48-f85969496c9d" containerName="istio-proxy" Nov 01 22:42:21.335534 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.335514 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="04419d55-67d8-420d-8f48-f85969496c9d" containerName="istio-proxy" Nov 01 22:42:21.345463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.342968 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-data\") pod \"04419d55-67d8-420d-8f48-f85969496c9d\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " Nov 01 22:42:21.345463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.343041 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-kube-api-access-4wbjd\" (UniqueName: \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-kube-api-access-4wbjd\") pod \"04419d55-67d8-420d-8f48-f85969496c9d\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " Nov 01 22:42:21.345463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.343184 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-socket\") pod \"04419d55-67d8-420d-8f48-f85969496c9d\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " Nov 01 22:42:21.345463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.343255 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-istio-token\" (UniqueName: \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-istio-token\") pod \"04419d55-67d8-420d-8f48-f85969496c9d\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " Nov 01 22:42:21.345463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.343304 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04419d55-67d8-420d-8f48-f85969496c9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04419d55-67d8-420d-8f48-f85969496c9d-istio-podinfo\") pod \"04419d55-67d8-420d-8f48-f85969496c9d\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " Nov 01 22:42:21.345463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.343360 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-credential-socket\") pod \"04419d55-67d8-420d-8f48-f85969496c9d\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " Nov 01 22:42:21.345463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.343615 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04419d55-67d8-420d-8f48-f85969496c9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04419d55-67d8-420d-8f48-f85969496c9d-istiod-ca-cert\") pod \"04419d55-67d8-420d-8f48-f85969496c9d\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " Nov 01 22:42:21.345463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.343740 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-envoy\") pod \"04419d55-67d8-420d-8f48-f85969496c9d\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " Nov 01 22:42:21.345463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.343789 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-certs\") pod \"04419d55-67d8-420d-8f48-f85969496c9d\" (UID: \"04419d55-67d8-420d-8f48-f85969496c9d\") " Nov 01 22:42:21.369490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.367523 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.373285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.370336 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-credential-socket" pod "04419d55-67d8-420d-8f48-f85969496c9d" (UID: "04419d55-67d8-420d-8f48-f85969496c9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:21.373285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.370473 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-data" pod "04419d55-67d8-420d-8f48-f85969496c9d" (UID: "04419d55-67d8-420d-8f48-f85969496c9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:21.373285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.370892 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-certs" pod "04419d55-67d8-420d-8f48-f85969496c9d" (UID: "04419d55-67d8-420d-8f48-f85969496c9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:21.373285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.371315 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04419d55-67d8-420d-8f48-f85969496c9d-istiod-ca-cert" pod "04419d55-67d8-420d-8f48-f85969496c9d" (UID: "04419d55-67d8-420d-8f48-f85969496c9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:21.373285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.373186 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-istio-token" pod "04419d55-67d8-420d-8f48-f85969496c9d" (UID: "04419d55-67d8-420d-8f48-f85969496c9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:21.375056 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.374734 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:21.378439 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.377423 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04419d55-67d8-420d-8f48-f85969496c9d-istio-podinfo" pod "04419d55-67d8-420d-8f48-f85969496c9d" (UID: "04419d55-67d8-420d-8f48-f85969496c9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:42:21.382691 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.382143 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg"] Nov 01 22:42:21.384240 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.383942 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-kube-api-access-4wbjd" pod "04419d55-67d8-420d-8f48-f85969496c9d" (UID: "04419d55-67d8-420d-8f48-f85969496c9d"). InnerVolumeSpecName "kube-api-access-4wbjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:21.384240 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.384233 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-socket" pod "04419d55-67d8-420d-8f48-f85969496c9d" (UID: "04419d55-67d8-420d-8f48-f85969496c9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:21.390844 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.390549 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ee8d1a-openshift-default-dockercfg-hsk87\"" Nov 01 22:42:21.413414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.413351 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-envoy" pod "04419d55-67d8-420d-8f48-f85969496c9d" (UID: "04419d55-67d8-420d-8f48-f85969496c9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:21.450272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.450233 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.450272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.450269 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04419d55-67d8-420d-8f48-f85969496c9d-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.450514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.450284 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.450514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.450298 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.450514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.450313 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.450514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.450327 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wbjd\" (UniqueName: \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-kube-api-access-4wbjd\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.450514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.450345 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04419d55-67d8-420d-8f48-f85969496c9d-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.450514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.450358 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04419d55-67d8-420d-8f48-f85969496c9d-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.450514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.450371 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04419d55-67d8-420d-8f48-f85969496c9d-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.578132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.565907 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.578132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.566001 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nmkc\" (UniqueName: \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-kube-api-access-9nmkc\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.578132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.566069 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.578132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.566326 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.578132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.566427 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.578132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.566575 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.578132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.566606 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.578132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.566767 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.578132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.566823 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.636514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.636484 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:42:21.674130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.674075 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.674326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.674194 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.674326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.674268 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.674457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.674385 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.674457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.674415 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.674615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.674527 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.674615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.674570 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.681177 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.674743 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.681318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.681250 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nmkc\" (UniqueName: \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-kube-api-access-9nmkc\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.681467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.676945 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.681544 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.678909 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.681544 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.679535 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.681544 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.680400 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.681544 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.676722 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.686793 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.686586 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.689548 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.689506 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz"] Nov 01 22:42:21.690590 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.690568 6466 cpu_manager.go:414] "containerMap: removing container" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" Nov 01 22:42:21.690590 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.690588 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" Nov 01 22:42:21.690803 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.690790 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerName="istio-proxy" Nov 01 22:42:21.702420 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.702391 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.703429 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.703084 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.704654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.704623 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nmkc\" (UniqueName: \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-kube-api-access-9nmkc\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:21.709244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.709075 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz"] Nov 01 22:42:21.709244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.709205 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.766454 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.766421 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9"] Nov 01 22:42:21.774653 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.772080 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:21.774653 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.772191 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:21.774653 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.772240 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:21.802059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.787928 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-token\" (UniqueName: \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-token\") pod \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " Nov 01 22:42:21.802059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.788079 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-podinfo\") pod \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.841502 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-token" pod "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" (UID: "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.841777 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-envoy\") pod \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.841842 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-data\") pod \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.841886 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-credential-socket\") pod \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.842005 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-socket\") pod \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.842148 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-certs\") pod \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.842235 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-kube-api-access-csqlq\" (UniqueName: \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-kube-api-access-csqlq\") pod \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.842413 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-credential-socket" pod "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" (UID: "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.842471 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istiod-ca-cert\") pod \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\" (UID: \"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4\") " Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.842742 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-socket" pod "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" (UID: "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.842776 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-data" pod "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" (UID: "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.842963 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.843053 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.843165 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istiod-ca-cert" pod "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" (UID: "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.843397 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.843412 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-certs" pod "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" (UID: "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.844909 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.844933 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.844989 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcnm9\" (UniqueName: \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-kube-api-access-hcnm9\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.845055 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.845070 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.845170 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.845344 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.845354 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.845364 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.845373 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.845382 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.845391 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.846062 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-envoy" pod "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" (UID: "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.846094 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-kube-api-access-csqlq" pod "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" (UID: "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4"). InnerVolumeSpecName "kube-api-access-csqlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:21.848032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.846146 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-podinfo" pod "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" (UID: "9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:42:21.854676 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.854638 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" event={"ID":"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1","Type":"ContainerStarted","Data":"ed87d02f50561f507f0060a89e6d6c2d691306b70e720be117e2cf153293ea26"} Nov 01 22:42:21.856707 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.856680 6466 generic.go:356] "Generic (PLEG): container finished" podID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" containerID="512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55" exitCode=0 Nov 01 22:42:21.856818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.856763 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" event={"ID":"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4","Type":"ContainerDied","Data":"512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55"} Nov 01 22:42:21.856886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.856827 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" event={"ID":"9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4","Type":"ContainerDied","Data":"8fe312bb84a32e7a0aa42d3ef7c7920610cf08ce74f4e4af6769aa13e54ddd93"} Nov 01 22:42:21.856886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.856857 6466 scope.go:122] "RemoveContainer" containerID="512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55" Nov 01 22:42:21.858220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.858196 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z" Nov 01 22:42:21.860763 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.860736 6466 generic.go:356] "Generic (PLEG): container finished" podID="04419d55-67d8-420d-8f48-f85969496c9d" containerID="0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50" exitCode=0 Nov 01 22:42:21.860854 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.860832 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" Nov 01 22:42:21.863911 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.863882 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" event={"ID":"04419d55-67d8-420d-8f48-f85969496c9d","Type":"ContainerDied","Data":"0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50"} Nov 01 22:42:21.864296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.863945 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4" event={"ID":"04419d55-67d8-420d-8f48-f85969496c9d","Type":"ContainerDied","Data":"df08251c3009f59005902395879bc0fd58bc281fd6ba9b13fa9acee627ba1936"} Nov 01 22:42:21.864296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.864022 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv" Nov 01 22:42:21.883624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.883600 6466 scope.go:122] "RemoveContainer" containerID="512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55" Nov 01 22:42:21.884062 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:42:21.884030 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55\": container with ID starting with 512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55 not found: ID does not exist" containerID="512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55" Nov 01 22:42:21.884201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.884075 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55"} err="failed to get container status \"512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55\": rpc error: code = NotFound desc = could not find container \"512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55\": container with ID starting with 512aa0ab53d9e0a546fe168fe3e8f0eb9275e1fdf40c76ae2bf127dad97edb55 not found: ID does not exist" Nov 01 22:42:21.884201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.884099 6466 scope.go:122] "RemoveContainer" containerID="0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50" Nov 01 22:42:21.912787 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.911274 6466 scope.go:122] "RemoveContainer" containerID="0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50" Nov 01 22:42:21.912888 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:42:21.912754 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50\": container with ID starting with 0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50 not found: ID does not exist" containerID="0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50" Nov 01 22:42:21.912888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.912810 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50"} err="failed to get container status \"0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50\": rpc error: code = NotFound desc = could not find container \"0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50\": container with ID starting with 0d1cf533f726b2d5dea23a8f072996c625a400e28f1f580c01b63617be535d50 not found: ID does not exist" Nov 01 22:42:21.913764 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.913743 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z"] Nov 01 22:42:21.918703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.918682 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-mkc4z"] Nov 01 22:42:21.935824 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.935772 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv"] Nov 01 22:42:21.941780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.941752 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vrgv"] Nov 01 22:42:21.947731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.947698 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.947893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.947763 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.947893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.947840 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.947893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.947880 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.948184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.947896 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.948184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.947946 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcnm9\" (UniqueName: \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-kube-api-access-hcnm9\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.948184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.948005 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.948184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.948023 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.948184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.948064 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.948535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.948306 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.948535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.948326 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.948535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.948342 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csqlq\" (UniqueName: \"kubernetes.io/projected/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4-kube-api-access-csqlq\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.948650 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.948576 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.948650 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.948631 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.948882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.948864 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.949143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.949121 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.949587 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.949563 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.950968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.950944 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.955534 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.955195 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4"] Nov 01 22:42:21.959641 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.959585 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.960805 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.960330 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-z2cj4"] Nov 01 22:42:21.966393 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.966365 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcnm9\" (UniqueName: \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-kube-api-access-hcnm9\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:21.967243 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:21.967220 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:22.004786 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.004499 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:22.154189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.152066 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:22.463952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.458964 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04419d55-67d8-420d-8f48-f85969496c9d" path="/var/lib/kubelet/pods/04419d55-67d8-420d-8f48-f85969496c9d/volumes" Nov 01 22:42:22.463952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.459923 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131d0e2f-d0ba-4623-8bdb-e7b969c526db" path="/var/lib/kubelet/pods/131d0e2f-d0ba-4623-8bdb-e7b969c526db/volumes" Nov 01 22:42:22.463952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.460679 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4" path="/var/lib/kubelet/pods/9bdd3ac7-20a2-4844-a99f-6304dc4a5bb4/volumes" Nov 01 22:42:22.610648 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.596280 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg"] Nov 01 22:42:22.633779 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.633735 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:22.633921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.633854 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:22.633921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.633913 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:22.787567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.787536 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz"] Nov 01 22:42:22.790522 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:42:22.790492 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbf3aa0_4a0b_4069_b219_6f73873bf995.slice/crio-b824392195915f30f35bef044aaf67178045bc132ac3a59d39fb92f07a4fbabf WatchSource:0}: Error finding container b824392195915f30f35bef044aaf67178045bc132ac3a59d39fb92f07a4fbabf: Status 404 returned error can't find the container with id b824392195915f30f35bef044aaf67178045bc132ac3a59d39fb92f07a4fbabf Nov 01 22:42:22.794698 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.794665 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:22.794810 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.794748 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:22.794810 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.794798 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:22.897478 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.897436 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" event={"ID":"3dbf3aa0-4a0b-4069-b219-6f73873bf995","Type":"ContainerStarted","Data":"b824392195915f30f35bef044aaf67178045bc132ac3a59d39fb92f07a4fbabf"} Nov 01 22:42:22.900664 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.900401 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" event={"ID":"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1","Type":"ContainerStarted","Data":"06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305"} Nov 01 22:42:22.906706 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.906685 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" event={"ID":"84e4b70b-8d5e-4d95-9a62-3996d9fac057","Type":"ContainerStarted","Data":"a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8"} Nov 01 22:42:22.907297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.907272 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" event={"ID":"84e4b70b-8d5e-4d95-9a62-3996d9fac057","Type":"ContainerStarted","Data":"df278f54efb9f68d9c05a2a22c2c8db9a1aad2c293c0e9883d71abb02c72d12e"} Nov 01 22:42:22.925678 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:22.925619 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podStartSLOduration=30.925598508 podStartE2EDuration="30.925598508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:22.921946899 +0000 UTC m=+1146.706385821" watchObservedRunningTime="2026-11-01 22:42:22.925598508 +0000 UTC m=+1146.710037430" Nov 01 22:42:23.007560 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:23.007470 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:23.008395 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:23.008357 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:23.008508 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:23.008411 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:23.375664 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:23.375632 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:23.377400 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:23.377374 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:23.377519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:23.377423 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:23.922486 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:23.922446 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" event={"ID":"3dbf3aa0-4a0b-4069-b219-6f73873bf995","Type":"ContainerStarted","Data":"ede69a7260c2703129e4749bc566c595b8fc73843a03869a4bb1dd7256224d1b"} Nov 01 22:42:23.946743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:23.946678 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podStartSLOduration=4.946656087 podStartE2EDuration="4.946656087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:22.952054186 +0000 UTC m=+1146.736493109" watchObservedRunningTime="2026-11-01 22:42:23.946656087 +0000 UTC m=+1147.731095009" Nov 01 22:42:23.947954 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:23.947908 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podStartSLOduration=4.947893736 podStartE2EDuration="4.947893736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:23.946198201 +0000 UTC m=+1147.730637123" watchObservedRunningTime="2026-11-01 22:42:23.947893736 +0000 UTC m=+1147.732332658" Nov 01 22:42:24.011178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:24.010546 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:24.011178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:24.010610 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:24.179665 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:24.179571 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:24.182130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:24.182072 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:24.182557 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:24.182498 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:24.375454 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:24.375412 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:24.375658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:24.375482 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:25.006511 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:25.006480 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:25.006901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:25.006531 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:25.176912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:25.176882 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:25.177086 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:25.176935 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:25.376025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:25.375946 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:25.376025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:25.376002 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:25.706413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:25.706383 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:42:26.007239 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:26.007150 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:26.007239 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:26.007207 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:26.176963 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:26.176928 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:26.177190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:26.176984 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:26.398013 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:26.397968 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:26.398233 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:26.398036 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:27.006548 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:27.006518 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:27.006729 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:27.006578 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:27.188133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:27.177486 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:27.188133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:27.177545 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:27.375971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:27.375892 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:27.375971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:27.375945 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:28.006691 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:28.006657 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:28.006888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:28.006712 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:28.176572 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:28.176541 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:28.176759 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:28.176596 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:28.375504 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:28.375413 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:28.375504 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:28.375470 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:29.021131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:29.018102 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:29.021131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:29.018211 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:29.176785 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:29.176747 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:29.176955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:29.176801 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:29.376177 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:29.376066 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:29.376177 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:29.376163 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:30.008751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:30.008710 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:30.012792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:30.012750 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:30.177543 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:30.177507 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:30.177719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:30.177560 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:30.376042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:30.375966 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:30.376042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:30.376018 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:31.007135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:31.006839 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:31.007135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:31.006903 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:31.177281 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:31.177246 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:31.177452 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:31.177312 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:31.375917 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:31.375839 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:31.375917 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:31.375901 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:32.007726 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:32.007692 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:32.008441 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:32.008411 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:32.181895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:32.181465 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:32.181895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:32.181528 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:32.375633 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:32.375550 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:32.375893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:32.375869 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:33.006857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:33.006822 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:33.007042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:33.006894 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:33.177053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:33.177012 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:33.177509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:33.177078 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:33.376623 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:33.376275 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:33.376623 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:33.376355 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:34.007201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:34.007165 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:34.007381 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:34.007220 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:34.177512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:34.177474 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:34.178572 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:34.178538 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:34.376280 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:34.376191 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:34.376280 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:34.376261 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:35.006553 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:35.006513 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:35.006760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:35.006569 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:35.193182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:35.193082 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:35.194474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:35.194412 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:35.376493 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:35.376405 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:35.376493 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:35.376473 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:36.007333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:36.007303 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:36.007514 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:36.007354 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:36.184132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:36.177549 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:36.184132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:36.177629 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:36.375273 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:36.375189 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:36.375780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:36.375407 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:36.375780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:36.375465 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:37.006353 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:37.006317 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:37.006592 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:37.006568 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:37.006679 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:37.006635 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:37.176660 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:37.176625 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:37.176902 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:37.176882 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:37.176955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:37.176938 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:37.375785 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:37.375700 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:37.375785 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:37.375749 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:38.007076 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:38.007043 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:38.007323 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:38.007121 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:38.176586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:38.176552 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:38.176787 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:38.176608 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:38.376574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:38.376494 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:38.377235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:38.377206 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:39.020235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:39.008552 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:39.020235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:39.008619 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:39.177182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:39.177144 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:39.177340 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:39.177208 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:39.376298 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:39.376208 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:39.376298 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:39.376279 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:40.006832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:40.006793 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:40.007342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:40.006864 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:40.176936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:40.176899 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:40.177182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:40.176962 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:40.380244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:40.375431 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:40.380244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:40.375506 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:41.007044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:41.006866 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:41.007044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:41.006927 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:41.184175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:41.183549 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:41.184175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:41.183613 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:41.410133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:41.403615 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:41.410133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:41.403678 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:42.020134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:42.014536 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:42.020134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:42.014615 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:42.180901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:42.180506 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:42.180901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:42.180571 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:42.288528 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:42.288418 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:42.377532 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:42.377499 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:42.377937 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:42.377906 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:42.556740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:42.556630 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:42.596390 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:42.593490 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:43.012952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:43.012555 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:43.012952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:43.012621 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:43.088819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:43.088356 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:43.186523 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:43.186486 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:43.186874 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:43.186844 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:43.376184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:43.375514 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:43.376184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:43.375580 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:44.018132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:44.010262 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:44.018132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:44.010321 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:44.509132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:44.508556 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:44.509132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:44.508620 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:44.663133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:44.660522 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:44.663133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:44.660580 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:45.024138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.007654 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:45.024138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.007725 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:45.189132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.188413 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:45.189132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.188507 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:45.312635 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.312538 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:45.327769 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.327172 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:45.427198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.423096 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:45.427198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.423186 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:45.635189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.632488 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:42:45.735164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.730197 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:42:45.961147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.950041 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2"] Nov 01 22:42:45.961147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.950384 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" containerID="cri-o://b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484" gracePeriod=30 Nov 01 22:42:46.032770 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:45.997865 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:46.212201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.185067 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:46.212201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.185158 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:46.212201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.191188 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:46.265173 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.216876 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:46.265173 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.216950 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:46.353184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.329211 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:46.353184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.334770 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:42:46.353184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.336489 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:42:46.487123 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.486851 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:46.487123 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.486912 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:46.653145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.645496 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:46.803341 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.803216 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:42:46.948968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.893604 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:46.948968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.893675 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:46.948968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:46.935757 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:47.217431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:47.048951 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:42:47.217431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:47.049528 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:42:47.217431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:47.050016 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:42:47.217431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:47.064190 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:47.217431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:47.064242 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:47.217431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:47.064782 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:42:47.266362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:47.266324 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:47.266632 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:47.266601 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:47.306357 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:47.306317 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:42:47.742465 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:47.736788 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:42:47.896411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:47.896360 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:48.021535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:48.010972 6466 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:42:48.021535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:48.011036 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 01 22:42:48.021535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:48.018849 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:48.021535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:48.018900 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:48.192216 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:48.192179 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 01 22:42:48.192553 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:48.192507 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 01 22:42:48.524301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:48.524267 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:48.529065 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:48.529036 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:42:48.535244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:48.535215 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:42:49.013605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:49.013561 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:42:49.015506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:49.015469 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 22:42:49.183592 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:49.183559 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:49.185472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:49.185447 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:42:49.336257 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:42:49.336222 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:42:50.012919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:50.012878 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:50.014527 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:50.014505 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:42:51.534726 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.534684 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:42:51.574718 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.574677 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x"] Nov 01 22:42:51.576278 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.576253 6466 cpu_manager.go:414] "containerMap: removing container" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" Nov 01 22:42:51.576386 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.576281 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" Nov 01 22:42:51.576626 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.576608 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerName="istio-proxy" Nov 01 22:42:51.609860 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.609630 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x"] Nov 01 22:42:51.609860 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.609763 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.688987 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.688952 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-podinfo\") pod \"e25203e8-2d60-4abd-93ec-49a19a6000bc\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " Nov 01 22:42:51.689262 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.689078 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-credential-socket\") pod \"e25203e8-2d60-4abd-93ec-49a19a6000bc\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " Nov 01 22:42:51.689262 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.689150 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-socket\") pod \"e25203e8-2d60-4abd-93ec-49a19a6000bc\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " Nov 01 22:42:51.689262 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.689226 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-kube-api-access-mfm5k\" (UniqueName: \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-kube-api-access-mfm5k\") pod \"e25203e8-2d60-4abd-93ec-49a19a6000bc\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " Nov 01 22:42:51.689432 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.689343 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-certs\") pod \"e25203e8-2d60-4abd-93ec-49a19a6000bc\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " Nov 01 22:42:51.689485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.689446 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e25203e8-2d60-4abd-93ec-49a19a6000bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e25203e8-2d60-4abd-93ec-49a19a6000bc-istiod-ca-cert\") pod \"e25203e8-2d60-4abd-93ec-49a19a6000bc\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " Nov 01 22:42:51.689545 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.689529 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-envoy\") pod \"e25203e8-2d60-4abd-93ec-49a19a6000bc\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " Nov 01 22:42:51.689638 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.689622 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-token\" (UniqueName: \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-token\") pod \"e25203e8-2d60-4abd-93ec-49a19a6000bc\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " Nov 01 22:42:51.689704 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.689690 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-data\") pod \"e25203e8-2d60-4abd-93ec-49a19a6000bc\" (UID: \"e25203e8-2d60-4abd-93ec-49a19a6000bc\") " Nov 01 22:42:51.690205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690091 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.690344 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690210 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330ac018-be0b-4502-9562-b0999cdc8245-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.690344 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690325 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.690453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690361 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330ac018-be0b-4502-9562-b0999cdc8245-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.690530 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690515 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.690586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690561 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.690644 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690608 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4h8q\" (UniqueName: \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-kube-api-access-r4h8q\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.690698 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690671 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-credential-socket" pod "e25203e8-2d60-4abd-93ec-49a19a6000bc" (UID: "e25203e8-2d60-4abd-93ec-49a19a6000bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:51.691095 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690785 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.691095 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690856 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.691095 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690895 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-certs" pod "e25203e8-2d60-4abd-93ec-49a19a6000bc" (UID: "e25203e8-2d60-4abd-93ec-49a19a6000bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:51.691095 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.690958 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:51.691371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.691183 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-socket" pod "e25203e8-2d60-4abd-93ec-49a19a6000bc" (UID: "e25203e8-2d60-4abd-93ec-49a19a6000bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:51.691371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.691254 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25203e8-2d60-4abd-93ec-49a19a6000bc-istiod-ca-cert" pod "e25203e8-2d60-4abd-93ec-49a19a6000bc" (UID: "e25203e8-2d60-4abd-93ec-49a19a6000bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:51.691371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.691354 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-data" pod "e25203e8-2d60-4abd-93ec-49a19a6000bc" (UID: "e25203e8-2d60-4abd-93ec-49a19a6000bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:51.693383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.693361 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-token" pod "e25203e8-2d60-4abd-93ec-49a19a6000bc" (UID: "e25203e8-2d60-4abd-93ec-49a19a6000bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:51.693637 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.693610 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-envoy" pod "e25203e8-2d60-4abd-93ec-49a19a6000bc" (UID: "e25203e8-2d60-4abd-93ec-49a19a6000bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:51.693748 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.693648 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-podinfo" pod "e25203e8-2d60-4abd-93ec-49a19a6000bc" (UID: "e25203e8-2d60-4abd-93ec-49a19a6000bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:42:51.693919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.693902 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-kube-api-access-mfm5k" pod "e25203e8-2d60-4abd-93ec-49a19a6000bc" (UID: "e25203e8-2d60-4abd-93ec-49a19a6000bc"). InnerVolumeSpecName "kube-api-access-mfm5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:51.792015 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.791977 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792240 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792025 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792240 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792058 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4h8q\" (UniqueName: \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-kube-api-access-r4h8q\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792240 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792140 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792240 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792182 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792353 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792433 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330ac018-be0b-4502-9562-b0999cdc8245-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792514 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792538 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792543 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792732 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792579 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792732 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792602 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330ac018-be0b-4502-9562-b0999cdc8245-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.792921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792762 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:51.792921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792779 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:51.792921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792791 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:51.792921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792804 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e25203e8-2d60-4abd-93ec-49a19a6000bc-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:51.792921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792819 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:51.792921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792834 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfm5k\" (UniqueName: \"kubernetes.io/projected/e25203e8-2d60-4abd-93ec-49a19a6000bc-kube-api-access-mfm5k\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:51.792921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792851 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e25203e8-2d60-4abd-93ec-49a19a6000bc-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:51.792921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792860 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e25203e8-2d60-4abd-93ec-49a19a6000bc-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:51.792921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.792857 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.793394 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.793239 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330ac018-be0b-4502-9562-b0999cdc8245-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.795040 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.795007 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.795193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.795081 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330ac018-be0b-4502-9562-b0999cdc8245-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.801169 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.801148 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.801507 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.801477 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4h8q\" (UniqueName: \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-kube-api-access-r4h8q\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:51.877936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.877889 6466 generic.go:356] "Generic (PLEG): container finished" podID="e25203e8-2d60-4abd-93ec-49a19a6000bc" containerID="b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484" exitCode=0 Nov 01 22:42:51.878159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.878035 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" Nov 01 22:42:51.878284 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.878038 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" event={"ID":"e25203e8-2d60-4abd-93ec-49a19a6000bc","Type":"ContainerDied","Data":"b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484"} Nov 01 22:42:51.878427 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.878360 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2" event={"ID":"e25203e8-2d60-4abd-93ec-49a19a6000bc","Type":"ContainerDied","Data":"a838d49d223f88f5d2db50c0532a1ddac9bacdb3e61a58193b0c06242daa4c22"} Nov 01 22:42:51.878427 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.878407 6466 scope.go:122] "RemoveContainer" containerID="b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484" Nov 01 22:42:51.893851 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.893814 6466 scope.go:122] "RemoveContainer" containerID="b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484" Nov 01 22:42:51.894140 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:42:51.894117 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484\": container with ID starting with b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484 not found: ID does not exist" containerID="b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484" Nov 01 22:42:51.894228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.894151 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484"} err="failed to get container status \"b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484\": rpc error: code = NotFound desc = could not find container \"b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484\": container with ID starting with b771a48d781ea928dbbc8a1902cf93937fb30d27d4be8cf913711a6d70d56484 not found: ID does not exist" Nov 01 22:42:51.909984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.909954 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2"] Nov 01 22:42:51.914969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.914942 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-kb9q2"] Nov 01 22:42:51.965183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:51.961616 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:52.346022 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:52.345944 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e25203e8-2d60-4abd-93ec-49a19a6000bc" path="/var/lib/kubelet/pods/e25203e8-2d60-4abd-93ec-49a19a6000bc/volumes" Nov 01 22:42:52.378884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:52.378853 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x"] Nov 01 22:42:52.379181 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:42:52.379143 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330ac018_be0b_4502_9562_b0999cdc8245.slice/crio-ae9205f3d874247f5904200ad93aff3190a681dde631bde486ae167b8f239aed WatchSource:0}: Error finding container ae9205f3d874247f5904200ad93aff3190a681dde631bde486ae167b8f239aed: Status 404 returned error can't find the container with id ae9205f3d874247f5904200ad93aff3190a681dde631bde486ae167b8f239aed Nov 01 22:42:52.382746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:52.382717 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:52.382832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:52.382781 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:52.382832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:52.382810 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:53.348574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:52.852481 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp"] Nov 01 22:42:53.348574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:52.852731 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" containerID="cri-o://8ddb9378ff27bd1d913ca0a60b56480579290d1e5a94d2293e76b513ba526037" gracePeriod=30 Nov 01 22:42:53.348574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:52.910066 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" event={"ID":"330ac018-be0b-4502-9562-b0999cdc8245","Type":"ContainerStarted","Data":"4edc1e58707ec0d31e717b79207875d54725dd8d967879d007689fe353db237d"} Nov 01 22:42:53.348574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:52.910161 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" event={"ID":"330ac018-be0b-4502-9562-b0999cdc8245","Type":"ContainerStarted","Data":"ae9205f3d874247f5904200ad93aff3190a681dde631bde486ae167b8f239aed"} Nov 01 22:42:53.348574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:52.935810 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podStartSLOduration=33.935795395 podStartE2EDuration="33.935795395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:52.932012266 +0000 UTC m=+1176.716451224" watchObservedRunningTime="2026-11-01 22:42:52.935795395 +0000 UTC m=+1176.720234316" Nov 01 22:42:53.348574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:52.963571 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:42:53.922773 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:53.922742 6466 generic.go:356] "Generic (PLEG): container finished" podID="3e34de96-6bf5-411e-b72d-2010871edf08" containerID="f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360" exitCode=0 Nov 01 22:42:53.922953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:53.922827 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" event={"ID":"3e34de96-6bf5-411e-b72d-2010871edf08","Type":"ContainerDied","Data":"f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360"} Nov 01 22:42:53.923650 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:53.923631 6466 scope.go:122] "RemoveContainer" containerID="f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360" Nov 01 22:42:53.925559 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:53.925533 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:53.925647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:53.925603 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:53.925647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:53.925637 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:53.945277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:53.945245 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:53.945430 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:53.945313 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:53.964973 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:53.964936 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:42:53.965188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:53.964996 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:42:54.403900 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:42:54.403834 6466 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8221bb_bcff_4e2a_af5c_76f36b0a6477.slice/crio-conmon-a661736aecfbf2f3003ca2e286777d9b4362dfe43e99d7ef2dcad773b161b487.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:42:54.973584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:54.973177 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:42:54.973584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:54.973294 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:42:55.009132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.006430 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" event={"ID":"3e34de96-6bf5-411e-b72d-2010871edf08","Type":"ContainerStarted","Data":"b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603"} Nov 01 22:42:55.038626 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.038363 6466 generic.go:356] "Generic (PLEG): container finished" podID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerID="a661736aecfbf2f3003ca2e286777d9b4362dfe43e99d7ef2dcad773b161b487" exitCode=0 Nov 01 22:42:55.040213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.040023 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" event={"ID":"2f8221bb-bcff-4e2a-af5c-76f36b0a6477","Type":"ContainerDied","Data":"a661736aecfbf2f3003ca2e286777d9b4362dfe43e99d7ef2dcad773b161b487"} Nov 01 22:42:55.048197 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.040735 6466 scope.go:122] "RemoveContainer" containerID="a661736aecfbf2f3003ca2e286777d9b4362dfe43e99d7ef2dcad773b161b487" Nov 01 22:42:55.057732 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.056367 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:55.057732 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.056468 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:55.057732 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.056516 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:42:55.614671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.614642 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:42:55.615238 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.615215 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:42:55.615292 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.615266 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:42:55.963094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.963058 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:42:55.963296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:55.963147 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:42:56.048856 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:56.048775 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" event={"ID":"2f8221bb-bcff-4e2a-af5c-76f36b0a6477","Type":"ContainerStarted","Data":"dc9af7de362fa4ad2b33abcd6ba4a74188bdd09692999cb35b550ffb60e97c1d"} Nov 01 22:42:56.614971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:56.614940 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:42:56.615416 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:56.614988 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:42:56.784184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:56.784146 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:42:56.784384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:56.784357 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:42:56.784464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:56.784412 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:42:56.962476 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:56.962440 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:42:56.962647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:56.962498 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:42:57.625132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:57.614150 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:42:57.625132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:57.614211 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:42:57.784496 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:57.784012 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:42:57.784496 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:57.784071 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:42:57.964044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:57.964012 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:42:57.964336 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:57.964292 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:42:58.161622 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.153073 6466 generic.go:356] "Generic (PLEG): container finished" podID="2417aba7-f7aa-49df-900b-659dfdfff057" containerID="8ddb9378ff27bd1d913ca0a60b56480579290d1e5a94d2293e76b513ba526037" exitCode=0 Nov 01 22:42:58.161622 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.153169 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" event={"ID":"2417aba7-f7aa-49df-900b-659dfdfff057","Type":"ContainerDied","Data":"8ddb9378ff27bd1d913ca0a60b56480579290d1e5a94d2293e76b513ba526037"} Nov 01 22:42:58.481022 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.480996 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:42:58.517583 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.517554 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg"] Nov 01 22:42:58.518501 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.518485 6466 cpu_manager.go:414] "containerMap: removing container" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" Nov 01 22:42:58.518553 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.518503 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" Nov 01 22:42:58.518645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.518636 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" containerName="istio-proxy" Nov 01 22:42:58.521949 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.521925 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.536441 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.536362 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9b7320-serving-cert\"" Nov 01 22:42:58.544796 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.544768 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg"] Nov 01 22:42:58.572404 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.572368 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-data\") pod \"2417aba7-f7aa-49df-900b-659dfdfff057\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " Nov 01 22:42:58.572592 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.572419 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-envoy\") pod \"2417aba7-f7aa-49df-900b-659dfdfff057\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " Nov 01 22:42:58.572592 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.572477 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-kube-api-access-9rwvb\" (UniqueName: \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-kube-api-access-9rwvb\") pod \"2417aba7-f7aa-49df-900b-659dfdfff057\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " Nov 01 22:42:58.572724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.572641 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-socket\") pod \"2417aba7-f7aa-49df-900b-659dfdfff057\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " Nov 01 22:42:58.572782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.572722 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-data" pod "2417aba7-f7aa-49df-900b-659dfdfff057" (UID: "2417aba7-f7aa-49df-900b-659dfdfff057"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:58.572869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.572838 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-certs\") pod \"2417aba7-f7aa-49df-900b-659dfdfff057\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " Nov 01 22:42:58.572985 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.572959 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-socket" pod "2417aba7-f7aa-49df-900b-659dfdfff057" (UID: "2417aba7-f7aa-49df-900b-659dfdfff057"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:58.573137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.573097 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-certs" pod "2417aba7-f7aa-49df-900b-659dfdfff057" (UID: "2417aba7-f7aa-49df-900b-659dfdfff057"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:58.573263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.573197 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-istio-token\" (UniqueName: \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-istio-token\") pod \"2417aba7-f7aa-49df-900b-659dfdfff057\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " Nov 01 22:42:58.573656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.573617 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2417aba7-f7aa-49df-900b-659dfdfff057-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2417aba7-f7aa-49df-900b-659dfdfff057-istiod-ca-cert\") pod \"2417aba7-f7aa-49df-900b-659dfdfff057\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " Nov 01 22:42:58.574151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.574016 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2417aba7-f7aa-49df-900b-659dfdfff057-istiod-ca-cert" pod "2417aba7-f7aa-49df-900b-659dfdfff057" (UID: "2417aba7-f7aa-49df-900b-659dfdfff057"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:58.574269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.574249 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2417aba7-f7aa-49df-900b-659dfdfff057-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2417aba7-f7aa-49df-900b-659dfdfff057-istio-podinfo\") pod \"2417aba7-f7aa-49df-900b-659dfdfff057\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " Nov 01 22:42:58.575097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.574738 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-credential-socket\") pod \"2417aba7-f7aa-49df-900b-659dfdfff057\" (UID: \"2417aba7-f7aa-49df-900b-659dfdfff057\") " Nov 01 22:42:58.575097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.575049 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-envoy" pod "2417aba7-f7aa-49df-900b-659dfdfff057" (UID: "2417aba7-f7aa-49df-900b-659dfdfff057"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:58.575097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.575078 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-credential-socket" pod "2417aba7-f7aa-49df-900b-659dfdfff057" (UID: "2417aba7-f7aa-49df-900b-659dfdfff057"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:42:58.575678 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.575651 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-kube-api-access-9rwvb" pod "2417aba7-f7aa-49df-900b-659dfdfff057" (UID: "2417aba7-f7aa-49df-900b-659dfdfff057"). InnerVolumeSpecName "kube-api-access-9rwvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:58.575988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.575970 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:58.576064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.575991 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:58.576064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.576000 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:58.576064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.576009 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rwvb\" (UniqueName: \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-kube-api-access-9rwvb\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:58.576064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.576019 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:58.576064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.576028 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2417aba7-f7aa-49df-900b-659dfdfff057-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:58.576064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.576036 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2417aba7-f7aa-49df-900b-659dfdfff057-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:58.576580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.576557 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-istio-token" pod "2417aba7-f7aa-49df-900b-659dfdfff057" (UID: "2417aba7-f7aa-49df-900b-659dfdfff057"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:58.579729 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.579700 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2417aba7-f7aa-49df-900b-659dfdfff057-istio-podinfo" pod "2417aba7-f7aa-49df-900b-659dfdfff057" (UID: "2417aba7-f7aa-49df-900b-659dfdfff057"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:42:58.614555 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.614522 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:42:58.614697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.614575 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:42:58.678203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.678165 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-metrics-certs\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.678584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.678286 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whlwj\" (UniqueName: \"kubernetes.io/projected/2bd09053-98bc-456a-9b80-9b32f584d2b1-kube-api-access-whlwj\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.678584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.678520 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bd09053-98bc-456a-9b80-9b32f584d2b1-openshift-service-ca\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.678584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.678545 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-maas-api-tls\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.678797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.678782 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2417aba7-f7aa-49df-900b-659dfdfff057-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:58.678834 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.678800 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2417aba7-f7aa-49df-900b-659dfdfff057-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:58.780157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.780123 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whlwj\" (UniqueName: \"kubernetes.io/projected/2bd09053-98bc-456a-9b80-9b32f584d2b1-kube-api-access-whlwj\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.780342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.780323 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bd09053-98bc-456a-9b80-9b32f584d2b1-openshift-service-ca\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.780386 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.780342 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-maas-api-tls\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.780492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.780478 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-metrics-certs\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.781004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.780980 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bd09053-98bc-456a-9b80-9b32f584d2b1-openshift-service-ca\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.782888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.782870 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-metrics-certs\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.783004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.782911 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-maas-api-tls\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.784528 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.784509 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:42:58.784584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.784551 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:42:58.790209 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.790153 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whlwj\" (UniqueName: \"kubernetes.io/projected/2bd09053-98bc-456a-9b80-9b32f584d2b1-kube-api-access-whlwj\") pod \"maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.842028 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.841996 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:42:58.962995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.962961 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:42:58.963223 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:58.963024 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:42:59.142524 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:42:59.142490 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd09053_98bc_456a_9b80_9b32f584d2b1.slice/crio-9e2bd4fa8d8765666dd59640872c0857a40035dc706248617cc233f91e72754e WatchSource:0}: Error finding container 9e2bd4fa8d8765666dd59640872c0857a40035dc706248617cc233f91e72754e: Status 404 returned error can't find the container with id 9e2bd4fa8d8765666dd59640872c0857a40035dc706248617cc233f91e72754e Nov 01 22:42:59.143238 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.143211 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg"] Nov 01 22:42:59.144170 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.144151 6466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:42:59.162857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.162829 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" event={"ID":"2bd09053-98bc-456a-9b80-9b32f584d2b1","Type":"ContainerStarted","Data":"9e2bd4fa8d8765666dd59640872c0857a40035dc706248617cc233f91e72754e"} Nov 01 22:42:59.164425 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.164403 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" Nov 01 22:42:59.164523 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.164424 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp" event={"ID":"2417aba7-f7aa-49df-900b-659dfdfff057","Type":"ContainerDied","Data":"ed5f8faa0c3cf46138311e01172319dd9bfd55143d9c6c7d485e753a278c84ca"} Nov 01 22:42:59.164523 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.164477 6466 scope.go:122] "RemoveContainer" containerID="8ddb9378ff27bd1d913ca0a60b56480579290d1e5a94d2293e76b513ba526037" Nov 01 22:42:59.195381 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.195346 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp"] Nov 01 22:42:59.200007 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.199980 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-gndlp"] Nov 01 22:42:59.614115 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.614076 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:42:59.614265 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.614144 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:42:59.791373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.789545 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:42:59.791373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.789607 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:42:59.966903 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.966834 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:42:59.967307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:42:59.967267 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:43:00.350128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:00.345727 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" event={"ID":"2bd09053-98bc-456a-9b80-9b32f584d2b1","Type":"ContainerStarted","Data":"23b19573cfd3bf67acc22d752d5e4f0a6f7c0db8194c235e824bbeb5ae24c382"} Nov 01 22:43:00.350128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:00.347084 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:43:00.425137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:00.420084 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" podStartSLOduration=32.038204582 podStartE2EDuration="32.420062s" totalImagesPullingTime="381.857418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:00.387830191 +0000 UTC m=+1184.172269154" watchObservedRunningTime="2026-11-01 22:43:00.420062 +0000 UTC m=+1184.204500924" Nov 01 22:43:00.486153 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:00.484979 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2417aba7-f7aa-49df-900b-659dfdfff057" path="/var/lib/kubelet/pods/2417aba7-f7aa-49df-900b-659dfdfff057/volumes" Nov 01 22:43:00.619192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:00.618028 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:43:00.619192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:00.618189 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:43:00.802848 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:00.797585 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:43:00.802848 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:00.797664 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:43:00.828133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:00.825211 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:43:01.015136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:00.975828 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:43:01.015136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:00.975886 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:43:01.628135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:01.627624 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:43:01.628135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:01.627694 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:43:01.749129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:01.744239 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.789127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:01.788550 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:43:01.789127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:01.788612 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:43:01.884191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:01.883408 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.914132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:01.909200 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.967132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:01.965614 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:43:01.967132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:01.965692 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:43:02.615784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:02.615732 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:43:02.635384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:02.635339 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:43:02.784703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:02.784639 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:43:02.784989 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:02.784957 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:43:02.963714 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:02.963674 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:43:02.964364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:02.964332 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:43:03.339185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:03.334311 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:03.660848 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:03.660810 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:43:03.661263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:03.661235 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:43:03.784611 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:03.784570 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:43:03.784923 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:03.784890 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:43:03.869264 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:03.869216 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:43:03.903252 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:03.903204 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:03.970093 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:03.969991 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:43:03.970739 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:03.970695 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:43:04.357197 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:04.356188 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:04.423983 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:04.423945 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:43:04.707198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:04.642590 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:43:04.707198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:04.642654 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:43:04.782136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:04.754464 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:04.832485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:04.828450 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:43:04.832485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:04.828511 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:43:04.934411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:04.909203 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:43:04.934411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:04.909710 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:43:04.982186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:04.971991 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:43:04.982186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:04.972052 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:43:05.405328 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:05.404207 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:05.632132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:05.618470 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:43:05.632132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:05.618537 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:43:05.928291 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:05.928246 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:43:05.950738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:05.950450 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:43:05.951360 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:05.951323 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:43:05.977781 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:05.928991 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:05.978447 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:05.968203 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:06.185195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.167544 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:43:06.185838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.185796 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:43:06.262132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.221617 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 01 22:43:06.418431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.409406 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:06.739533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.739496 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:43:06.739824 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.739780 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:43:06.766290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.766242 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:06.806340 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.806301 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:43:06.806638 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.806609 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:43:06.913133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.911313 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Nov 01 22:43:06.959132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.957436 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:43:06.964258 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.964225 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:43:06.964469 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.964442 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:43:06.964939 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.964917 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:43:06.978228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.978180 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:43:06.979010 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:06.978991 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:43:07.427142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.426779 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 01 22:43:07.645144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.630506 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:43:07.645144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.630565 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:43:07.731394 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.731300 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:43:07.771677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.771638 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:07.772501 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.772475 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:43:07.794148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.792015 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:43:07.794148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.792069 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:43:07.986216 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.980664 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:43:07.986216 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.983468 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:43:07.986216 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.985091 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:43:07.986216 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.985152 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:43:07.986216 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:07.985458 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:43:08.019624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:08.018893 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:43:08.439073 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:08.439028 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:08.486145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:08.473794 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:43:08.845191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:08.844210 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:43:08.845191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:08.844273 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 01 22:43:09.004734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.004674 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.006942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.006907 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:43:09.028224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.028181 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.082166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.073226 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:43:09.082166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.073285 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 01 22:43:09.124460 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.122337 6466 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.144428 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.144393 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:43:09.168004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.167955 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:43:09.597169 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.591753 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:43:09.615902 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.615867 6466 patch_prober.go:29] interesting pod/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:43:09.625880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.625841 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 22:43:09.645131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.632887 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:43:09.953528 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.953500 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:43:09.982631 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:09.982599 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:43:10.102728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:10.102698 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:43:10.110505 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:10.110479 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:43:10.618974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:10.618942 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:43:10.620143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:10.620122 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:43:15.250742 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.250709 6466 generic.go:356] "Generic (PLEG): container finished" podID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerID="64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e" exitCode=0 Nov 01 22:43:15.251249 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.250776 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" event={"ID":"f41e060e-e33f-478c-8019-ede84aeb4d13","Type":"ContainerDied","Data":"64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e"} Nov 01 22:43:15.251343 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.251330 6466 scope.go:122] "RemoveContainer" containerID="64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e" Nov 01 22:43:15.253395 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.253362 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:15.253490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.253440 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:15.253490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.253480 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:15.490878 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.490846 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6"] Nov 01 22:43:15.491080 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.491058 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" containerID="cri-o://c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088" gracePeriod=30 Nov 01 22:43:15.502918 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.502827 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp"] Nov 01 22:43:15.503065 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.503039 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" containerID="cri-o://e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc" gracePeriod=30 Nov 01 22:43:15.510019 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.509990 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn"] Nov 01 22:43:15.510202 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:15.510183 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" containerID="cri-o://c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443" gracePeriod=30 Nov 01 22:43:16.315672 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:16.315637 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" event={"ID":"f41e060e-e33f-478c-8019-ede84aeb4d13","Type":"ContainerStarted","Data":"b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78"} Nov 01 22:43:16.607650 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:16.607570 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:43:16.607801 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:16.607724 6466 patch_prober.go:29] interesting pod/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 22:43:16.607801 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:16.607780 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 22:43:17.608551 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:17.608515 6466 patch_prober.go:29] interesting pod/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 22:43:17.608984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:17.608583 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 22:43:18.608385 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:18.608351 6466 patch_prober.go:29] interesting pod/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 22:43:18.608575 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:18.608414 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 22:43:19.306244 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:43:19.306206 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:43:19.607882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:19.607801 6466 patch_prober.go:29] interesting pod/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 22:43:19.607882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:19.607857 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 22:43:20.607904 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.607870 6466 patch_prober.go:29] interesting pod/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 22:43:20.608316 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.607926 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 22:43:20.804453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.804415 6466 scope.go:122] "RemoveContainer" containerID="035d24c71c565cb8d1404240ff5377a54b982c344fa13bb21d62df809ddcfcbd" Nov 01 22:43:20.908130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.907836 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:43:20.940140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.935047 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-credential-socket\") pod \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " Nov 01 22:43:20.948158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.940463 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-socket\") pod \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " Nov 01 22:43:20.948158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.940532 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-podinfo\") pod \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " Nov 01 22:43:20.948158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.940579 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-token\" (UniqueName: \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-token\") pod \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " Nov 01 22:43:20.948158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.940648 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-kube-api-access-c6qrw\" (UniqueName: \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-kube-api-access-c6qrw\") pod \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " Nov 01 22:43:20.948158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.940706 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istiod-ca-cert\") pod \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " Nov 01 22:43:20.948158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.940799 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-data\") pod \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " Nov 01 22:43:20.948158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.940850 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-certs\") pod \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " Nov 01 22:43:20.948158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.941044 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-envoy\") pod \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\" (UID: \"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e\") " Nov 01 22:43:20.968148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.957015 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-data" pod "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" (UID: "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:20.968148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.957404 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istiod-ca-cert" pod "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" (UID: "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:20.968148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.957702 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-certs" pod "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" (UID: "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:20.968148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.957981 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-credential-socket" pod "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" (UID: "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:20.968148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.958040 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-socket" pod "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" (UID: "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:20.987090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.987026 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv"] Nov 01 22:43:20.992027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.991994 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" Nov 01 22:43:20.992267 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.992246 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" Nov 01 22:43:20.992739 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:20.992718 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerName="istio-proxy" Nov 01 22:43:21.002549 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.002511 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-podinfo" pod "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" (UID: "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:43:21.017550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.017198 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-kube-api-access-c6qrw" pod "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" (UID: "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e"). InnerVolumeSpecName "kube-api-access-c6qrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:21.050397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.047293 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-envoy" pod "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" (UID: "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:21.059741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.055678 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.059741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.055718 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.059741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.055734 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.059741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.055749 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6qrw\" (UniqueName: \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-kube-api-access-c6qrw\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.059741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.055764 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.059741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.055778 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.059741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.055791 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.059741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.055806 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.068175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.068101 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-token" pod "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" (UID: "d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:21.070585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.070552 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv"] Nov 01 22:43:21.070726 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.070707 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.072804 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.072773 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:43:21.220661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.214456 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-podinfo\") pod \"5fade718-419b-4d7a-9dd9-417cb6ec2621\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " Nov 01 22:43:21.220661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.214623 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-data\") pod \"5fade718-419b-4d7a-9dd9-417cb6ec2621\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " Nov 01 22:43:21.220661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.214677 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-token\" (UniqueName: \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-token\") pod \"5fade718-419b-4d7a-9dd9-417cb6ec2621\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " Nov 01 22:43:21.220661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.214765 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-kube-api-access-5ltn9\" (UniqueName: \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-kube-api-access-5ltn9\") pod \"5fade718-419b-4d7a-9dd9-417cb6ec2621\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " Nov 01 22:43:21.220661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.214932 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-socket\") pod \"5fade718-419b-4d7a-9dd9-417cb6ec2621\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " Nov 01 22:43:21.220661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.214963 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fade718-419b-4d7a-9dd9-417cb6ec2621-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fade718-419b-4d7a-9dd9-417cb6ec2621-istiod-ca-cert\") pod \"5fade718-419b-4d7a-9dd9-417cb6ec2621\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " Nov 01 22:43:21.220661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.215126 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-envoy\") pod \"5fade718-419b-4d7a-9dd9-417cb6ec2621\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " Nov 01 22:43:21.220661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.215191 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-credential-socket\") pod \"5fade718-419b-4d7a-9dd9-417cb6ec2621\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " Nov 01 22:43:21.220661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.215227 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-certs\") pod \"5fade718-419b-4d7a-9dd9-417cb6ec2621\" (UID: \"5fade718-419b-4d7a-9dd9-417cb6ec2621\") " Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.239201 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-data" pod "5fade718-419b-4d7a-9dd9-417cb6ec2621" (UID: "5fade718-419b-4d7a-9dd9-417cb6ec2621"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.239480 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-socket" pod "5fade718-419b-4d7a-9dd9-417cb6ec2621" (UID: "5fade718-419b-4d7a-9dd9-417cb6ec2621"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.243296 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fade718-419b-4d7a-9dd9-417cb6ec2621-istiod-ca-cert" pod "5fade718-419b-4d7a-9dd9-417cb6ec2621" (UID: "5fade718-419b-4d7a-9dd9-417cb6ec2621"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.245414 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.245581 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsfsv\" (UniqueName: \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-kube-api-access-rsfsv\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.245632 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.245693 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.245757 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.245832 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.245942 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.246060 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.265134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.246160 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.276162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.274195 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-certs" pod "5fade718-419b-4d7a-9dd9-417cb6ec2621" (UID: "5fade718-419b-4d7a-9dd9-417cb6ec2621"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:21.294735 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.276563 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-credential-socket" pod "5fade718-419b-4d7a-9dd9-417cb6ec2621" (UID: "5fade718-419b-4d7a-9dd9-417cb6ec2621"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:21.309312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.308099 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.309312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.308147 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.309312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.308165 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.309312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.308179 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fade718-419b-4d7a-9dd9-417cb6ec2621-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.309312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.308200 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.309312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.308235 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.403139 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.401950 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-kube-api-access-5ltn9" pod "5fade718-419b-4d7a-9dd9-417cb6ec2621" (UID: "5fade718-419b-4d7a-9dd9-417cb6ec2621"). InnerVolumeSpecName "kube-api-access-5ltn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:21.412658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.411604 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.412658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.411737 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsfsv\" (UniqueName: \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-kube-api-access-rsfsv\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.412658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.411778 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.412658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.411829 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.412658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.411885 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.412658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.411946 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.412658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.412058 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.412658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.412171 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.417548 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.417357 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.417869 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.418780 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.421269 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.422077 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7"] Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.422560 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.423133 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.423369 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ltn9\" (UniqueName: \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-kube-api-access-5ltn9\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.423467 6466 cpu_manager.go:414] "containerMap: removing container" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.423486 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.423519 6466 cpu_manager.go:414] "containerMap: removing container" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.423527 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.423802 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerName="istio-proxy" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.423820 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerName="istio-proxy" Nov 01 22:43:21.424172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.423871 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.425473 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.425423 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-envoy" pod "5fade718-419b-4d7a-9dd9-417cb6ec2621" (UID: "5fade718-419b-4d7a-9dd9-417cb6ec2621"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:21.425929 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.425894 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-podinfo" pod "5fade718-419b-4d7a-9dd9-417cb6ec2621" (UID: "5fade718-419b-4d7a-9dd9-417cb6ec2621"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:43:21.428401 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.428359 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-token" pod "5fade718-419b-4d7a-9dd9-417cb6ec2621" (UID: "5fade718-419b-4d7a-9dd9-417cb6ec2621"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:21.429032 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.428983 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.430014 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.429988 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsfsv\" (UniqueName: \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-kube-api-access-rsfsv\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.430977 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.430956 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.431842 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.431821 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.432973 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.432954 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7"] Nov 01 22:43:21.433084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.433041 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.467279 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.464081 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq"] Nov 01 22:43:21.481781 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.477969 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.488061 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.488034 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq"] Nov 01 22:43:21.517792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.517747 6466 generic.go:356] "Generic (PLEG): container finished" podID="3e843f55-141b-4fab-a772-fa84acc14d2e" containerID="c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443" exitCode=0 Nov 01 22:43:21.517972 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.517887 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" event={"ID":"3e843f55-141b-4fab-a772-fa84acc14d2e","Type":"ContainerDied","Data":"c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443"} Nov 01 22:43:21.518033 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.517984 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" event={"ID":"3e843f55-141b-4fab-a772-fa84acc14d2e","Type":"ContainerDied","Data":"9b67a6b3f021e1549c6377de16f2daae1866629fee9773e79411decadaa124ab"} Nov 01 22:43:21.518033 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.518025 6466 scope.go:122] "RemoveContainer" containerID="c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443" Nov 01 22:43:21.518293 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.518273 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn" Nov 01 22:43:21.521376 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.521343 6466 generic.go:356] "Generic (PLEG): container finished" podID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" containerID="e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc" exitCode=0 Nov 01 22:43:21.521671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.521647 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" event={"ID":"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e","Type":"ContainerDied","Data":"e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc"} Nov 01 22:43:21.523206 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.521710 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" event={"ID":"d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e","Type":"ContainerDied","Data":"89d99d7b4e32c83ef3473bca0a44de9651c0c39d2aebc7eba76bd0f40f72d1ac"} Nov 01 22:43:21.523206 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.521736 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp" Nov 01 22:43:21.523796 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.523767 6466 generic.go:356] "Generic (PLEG): container finished" podID="5fade718-419b-4d7a-9dd9-417cb6ec2621" containerID="c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088" exitCode=0 Nov 01 22:43:21.523947 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.523926 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" Nov 01 22:43:21.524121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.524080 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" event={"ID":"5fade718-419b-4d7a-9dd9-417cb6ec2621","Type":"ContainerDied","Data":"c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088"} Nov 01 22:43:21.524215 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.524158 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6" event={"ID":"5fade718-419b-4d7a-9dd9-417cb6ec2621","Type":"ContainerDied","Data":"c8ec0ce0ef57942ecb8c866a4605171b6ff0d485c406747e7a4cf248dd686d09"} Nov 01 22:43:21.525073 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.525043 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-data\") pod \"3e843f55-141b-4fab-a772-fa84acc14d2e\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " Nov 01 22:43:21.525227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.525198 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-certs\") pod \"3e843f55-141b-4fab-a772-fa84acc14d2e\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " Nov 01 22:43:21.525307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.525250 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-token\" (UniqueName: \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-token\") pod \"3e843f55-141b-4fab-a772-fa84acc14d2e\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " Nov 01 22:43:21.525368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.525309 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e843f55-141b-4fab-a772-fa84acc14d2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e843f55-141b-4fab-a772-fa84acc14d2e-istiod-ca-cert\") pod \"3e843f55-141b-4fab-a772-fa84acc14d2e\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " Nov 01 22:43:21.525466 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.525447 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-podinfo\") pod \"3e843f55-141b-4fab-a772-fa84acc14d2e\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " Nov 01 22:43:21.525608 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.525576 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-socket\") pod \"3e843f55-141b-4fab-a772-fa84acc14d2e\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " Nov 01 22:43:21.525678 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.525630 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-envoy\") pod \"3e843f55-141b-4fab-a772-fa84acc14d2e\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " Nov 01 22:43:21.525737 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.525675 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-certs" pod "3e843f55-141b-4fab-a772-fa84acc14d2e" (UID: "3e843f55-141b-4fab-a772-fa84acc14d2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:21.525846 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.525827 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-kube-api-access-rv9xt\" (UniqueName: \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-kube-api-access-rv9xt\") pod \"3e843f55-141b-4fab-a772-fa84acc14d2e\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " Nov 01 22:43:21.525918 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.525896 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-credential-socket\") pod \"3e843f55-141b-4fab-a772-fa84acc14d2e\" (UID: \"3e843f55-141b-4fab-a772-fa84acc14d2e\") " Nov 01 22:43:21.525981 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.525964 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-data" pod "3e843f55-141b-4fab-a772-fa84acc14d2e" (UID: "3e843f55-141b-4fab-a772-fa84acc14d2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:21.526735 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.526705 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wmrm\" (UniqueName: \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-kube-api-access-6wmrm\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.526848 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.526822 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.526955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.526936 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.527023 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.526994 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.527079 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527027 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.527079 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527054 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c48728-a545-4f28-a74a-99f701418e9d-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.527269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527086 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wmhg\" (UniqueName: \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-kube-api-access-5wmhg\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.527269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527178 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.527269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527218 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.527440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527330 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.527440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527372 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.527440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527427 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c48728-a545-4f28-a74a-99f701418e9d-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.527615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527504 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.527615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527544 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.527731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527668 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.527788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527752 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.527844 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527801 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.527903 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.527842 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.528048 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.528025 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.528220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.528054 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.528220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.528070 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.528220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.528086 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fade718-419b-4d7a-9dd9-417cb6ec2621-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.528220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.528122 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.531040 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.531002 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-kube-api-access-rv9xt" pod "3e843f55-141b-4fab-a772-fa84acc14d2e" (UID: "3e843f55-141b-4fab-a772-fa84acc14d2e"). InnerVolumeSpecName "kube-api-access-rv9xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:21.531436 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.531414 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-credential-socket" pod "3e843f55-141b-4fab-a772-fa84acc14d2e" (UID: "3e843f55-141b-4fab-a772-fa84acc14d2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:21.531893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.531596 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-socket" pod "3e843f55-141b-4fab-a772-fa84acc14d2e" (UID: "3e843f55-141b-4fab-a772-fa84acc14d2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:21.534351 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.534284 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-token" pod "3e843f55-141b-4fab-a772-fa84acc14d2e" (UID: "3e843f55-141b-4fab-a772-fa84acc14d2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:21.537187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.535179 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e843f55-141b-4fab-a772-fa84acc14d2e-istiod-ca-cert" pod "3e843f55-141b-4fab-a772-fa84acc14d2e" (UID: "3e843f55-141b-4fab-a772-fa84acc14d2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:21.537187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.535915 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-envoy" pod "3e843f55-141b-4fab-a772-fa84acc14d2e" (UID: "3e843f55-141b-4fab-a772-fa84acc14d2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:21.553028 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.552962 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-podinfo" pod "3e843f55-141b-4fab-a772-fa84acc14d2e" (UID: "3e843f55-141b-4fab-a772-fa84acc14d2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:43:21.616071 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.616035 6466 patch_prober.go:29] interesting pod/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 01 22:43:21.616744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.616709 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 01 22:43:21.665182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.639061 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.683407 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.655320 6466 scope.go:122] "RemoveContainer" containerID="c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.683542 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.672678 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.683580 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.683723 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c48728-a545-4f28-a74a-99f701418e9d-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.683842 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.683896 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.683932 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.684057 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.684197 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.684236 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.684267 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.693131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.684354 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wmrm\" (UniqueName: \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-kube-api-access-6wmrm\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.694212 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.694179 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.694474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.694458 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.707858 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.707830 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.708042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.708031 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.708141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.708124 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c48728-a545-4f28-a74a-99f701418e9d-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.708257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.708243 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wmhg\" (UniqueName: \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-kube-api-access-5wmhg\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.708334 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:43:21.708302 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443\": container with ID starting with c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443 not found: ID does not exist" containerID="c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443" Nov 01 22:43:21.708393 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.708351 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443"} err="failed to get container status \"c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443\": rpc error: code = NotFound desc = could not find container \"c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443\": container with ID starting with c971aadc482c38236b8d805adb7c48b09279ce8170e3a7c42965ca64bf447443 not found: ID does not exist" Nov 01 22:43:21.708393 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.708381 6466 scope.go:122] "RemoveContainer" containerID="e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc" Nov 01 22:43:21.708538 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.708511 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.709587 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.709565 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.709985 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.709965 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.713396 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.712046 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c48728-a545-4f28-a74a-99f701418e9d-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.714146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.713565 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.714146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.713745 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.714146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.713815 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.714146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.713853 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.714146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.713882 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e843f55-141b-4fab-a772-fa84acc14d2e-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.714146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.713952 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:21.714146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.714039 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.714582 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.714299 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.715077 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.714717 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.715077 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.714788 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.715077 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.714810 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.715077 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.714840 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.715077 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.714862 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv9xt\" (UniqueName: \"kubernetes.io/projected/3e843f55-141b-4fab-a772-fa84acc14d2e-kube-api-access-rv9xt\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.715077 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.714937 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e843f55-141b-4fab-a772-fa84acc14d2e-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:21.716093 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.715898 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c48728-a545-4f28-a74a-99f701418e9d-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.718145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.718090 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.720727 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.720696 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wmrm\" (UniqueName: \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-kube-api-access-6wmrm\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.721408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.721375 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.732836 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.732805 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wmhg\" (UniqueName: \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-kube-api-access-5wmhg\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.735699 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.735673 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.742141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.742091 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp"] Nov 01 22:43:21.756474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.756409 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-98xrp"] Nov 01 22:43:21.763579 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.763545 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6"] Nov 01 22:43:21.770608 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.770583 6466 scope.go:122] "RemoveContainer" containerID="e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc" Nov 01 22:43:21.770730 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.770697 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hjgr6"] Nov 01 22:43:21.771009 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:43:21.770977 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc\": container with ID starting with e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc not found: ID does not exist" containerID="e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc" Nov 01 22:43:21.771159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.771016 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc"} err="failed to get container status \"e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc\": rpc error: code = NotFound desc = could not find container \"e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc\": container with ID starting with e101cfedc382e8a5bfc9576ce05c8c417d7e03ad9bebbbe63dbd7d0ac0ec05dc not found: ID does not exist" Nov 01 22:43:21.771159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.771044 6466 scope.go:122] "RemoveContainer" containerID="c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088" Nov 01 22:43:21.788861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.785321 6466 scope.go:122] "RemoveContainer" containerID="c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088" Nov 01 22:43:21.788861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.785529 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:21.788861 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:43:21.785649 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088\": container with ID starting with c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088 not found: ID does not exist" containerID="c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088" Nov 01 22:43:21.788861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.785673 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088"} err="failed to get container status \"c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088\": rpc error: code = NotFound desc = could not find container \"c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088\": container with ID starting with c58c0d58e0f2d980d4194b41ce9ffcff9e2d7a4c9a1f0b250153dd76e8c8c088 not found: ID does not exist" Nov 01 22:43:21.811010 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.810963 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:21.861157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.860562 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn"] Nov 01 22:43:21.887846 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:21.887806 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-ktkbn"] Nov 01 22:43:22.085254 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.085227 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv"] Nov 01 22:43:22.091870 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.091834 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:22.091966 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.091908 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:22.091966 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.091939 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:22.313071 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.313045 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7"] Nov 01 22:43:22.313405 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:43:22.313382 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c48728_a545_4f28_a74a_99f701418e9d.slice/crio-b7ad3f6375496afc8c06487b0a80b8518d0a017cef0a6fc3c8b811689329863c WatchSource:0}: Error finding container b7ad3f6375496afc8c06487b0a80b8518d0a017cef0a6fc3c8b811689329863c: Status 404 returned error can't find the container with id b7ad3f6375496afc8c06487b0a80b8518d0a017cef0a6fc3c8b811689329863c Nov 01 22:43:22.318235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.318202 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:22.318318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.318285 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:22.318438 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.318332 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:22.322027 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:43:22.321998 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb948b12_1e45_4e5f_9750_eed6b5ad09cf.slice/crio-07579e79bae96233f83b469f08a79004cdc8d695718576039fa45d591240c347 WatchSource:0}: Error finding container 07579e79bae96233f83b469f08a79004cdc8d695718576039fa45d591240c347: Status 404 returned error can't find the container with id 07579e79bae96233f83b469f08a79004cdc8d695718576039fa45d591240c347 Nov 01 22:43:22.322294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.322276 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq"] Nov 01 22:43:22.325478 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.325449 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:22.325585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.325540 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:22.325646 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.325589 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:43:22.350498 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.350472 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e843f55-141b-4fab-a772-fa84acc14d2e" path="/var/lib/kubelet/pods/3e843f55-141b-4fab-a772-fa84acc14d2e/volumes" Nov 01 22:43:22.351120 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.351088 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fade718-419b-4d7a-9dd9-417cb6ec2621" path="/var/lib/kubelet/pods/5fade718-419b-4d7a-9dd9-417cb6ec2621/volumes" Nov 01 22:43:22.351855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.351833 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e" path="/var/lib/kubelet/pods/d8e7faaa-1491-4c8e-93b4-1e66c9fccd1e/volumes" Nov 01 22:43:22.551498 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.551447 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" event={"ID":"6691f77d-0336-4b20-a16f-9d8b5f2159e2","Type":"ContainerStarted","Data":"dc53a77f9fc6526d7974b64406175f2742531b105f6805db539b2ab96c174ce9"} Nov 01 22:43:22.551770 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.551725 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" event={"ID":"6691f77d-0336-4b20-a16f-9d8b5f2159e2","Type":"ContainerStarted","Data":"474921c12d9e7ed8540e67a58dc9ee838c368a244af7bee1aa8a6915e6842303"} Nov 01 22:43:22.554960 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.554248 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" event={"ID":"27c48728-a545-4f28-a74a-99f701418e9d","Type":"ContainerStarted","Data":"de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a"} Nov 01 22:43:22.554960 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.554306 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" event={"ID":"27c48728-a545-4f28-a74a-99f701418e9d","Type":"ContainerStarted","Data":"b7ad3f6375496afc8c06487b0a80b8518d0a017cef0a6fc3c8b811689329863c"} Nov 01 22:43:22.558690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.558666 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" event={"ID":"eb948b12-1e45-4e5f-9750-eed6b5ad09cf","Type":"ContainerStarted","Data":"5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb"} Nov 01 22:43:22.558784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.558710 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" event={"ID":"eb948b12-1e45-4e5f-9750-eed6b5ad09cf","Type":"ContainerStarted","Data":"07579e79bae96233f83b469f08a79004cdc8d695718576039fa45d591240c347"} Nov 01 22:43:22.586866 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.586816 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" podStartSLOduration=3.586802216 podStartE2EDuration="3.586802216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:22.581236003 +0000 UTC m=+1206.365674929" watchObservedRunningTime="2026-11-01 22:43:22.586802216 +0000 UTC m=+1206.371241137" Nov 01 22:43:22.614091 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.614016 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:43:22.614702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.614675 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:43:22.615975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.615942 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:43:22.616367 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.616327 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" podStartSLOduration=48.61631511 podStartE2EDuration="48.61631511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:22.608132394 +0000 UTC m=+1206.392571315" watchObservedRunningTime="2026-11-01 22:43:22.61631511 +0000 UTC m=+1206.400754032" Nov 01 22:43:22.636292 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.636227 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" podStartSLOduration=48.636208624 podStartE2EDuration="48.636208624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:22.631928461 +0000 UTC m=+1206.416367383" watchObservedRunningTime="2026-11-01 22:43:22.636208624 +0000 UTC m=+1206.420647547" Nov 01 22:43:22.714711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.714635 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:22.786152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.786119 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:22.811360 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:22.811326 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:23.715987 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:23.715954 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 22:43:23.716434 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:23.716000 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 22:43:23.787716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:23.787681 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:43:23.787881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:23.787738 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" podUID="27c48728-a545-4f28-a74a-99f701418e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 22:43:23.813435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:23.813401 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:43:23.813591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:23.813453 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 22:43:24.721594 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:24.721537 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 22:43:24.722311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:24.722280 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 22:43:24.788580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:24.788542 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:43:24.788893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:24.788859 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" podUID="27c48728-a545-4f28-a74a-99f701418e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 22:43:24.818135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:24.811908 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:43:24.818135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:24.811971 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 22:43:25.716135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:25.715489 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 22:43:25.716135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:25.715580 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 22:43:25.791179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:25.790742 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:43:25.791179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:25.790803 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" podUID="27c48728-a545-4f28-a74a-99f701418e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 22:43:25.812533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:25.812491 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:43:25.812748 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:25.812570 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 22:43:26.714875 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:26.714838 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 22:43:26.715195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:26.715168 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 22:43:26.787341 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:26.787306 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:43:26.787632 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:26.787593 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" podUID="27c48728-a545-4f28-a74a-99f701418e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 22:43:26.812718 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:26.812675 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:43:26.813327 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:26.813298 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 22:43:27.721364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:27.721320 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 22:43:27.721679 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:27.721642 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 22:43:27.861231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:27.861195 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:43:27.862379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:27.862344 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" podUID="27c48728-a545-4f28-a74a-99f701418e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 22:43:27.996000 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:27.995895 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:43:27.996347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:27.996284 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 22:43:28.759131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:28.738221 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 22:43:28.759131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:28.738299 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 22:43:28.836146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:28.834289 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:43:28.836146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:28.834374 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 22:43:28.864276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:28.864241 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:43:28.879268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:28.879208 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" podUID="27c48728-a545-4f28-a74a-99f701418e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 22:43:29.714722 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:29.714680 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 22:43:29.714917 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:29.714766 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 22:43:29.792980 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:29.792948 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:29.793449 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:29.793427 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:29.794142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:29.794083 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:43:29.810957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:29.810937 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:43:29.811060 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:29.810986 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 22:43:30.715379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:30.715343 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 22:43:30.715778 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:30.715397 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 22:43:30.811084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:30.811049 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:43:30.811286 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:30.811097 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 01 22:43:31.802265 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:31.800130 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:31.802265 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:31.801176 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:31.802997 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:31.802967 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:43:31.816466 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:31.816437 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:31.816745 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:31.816724 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:31.817498 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:31.817481 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:43:46.029225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:45.960165 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7021-aa3d9b"] Nov 01 22:43:46.085138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:46.080500 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7021-aa3d9b"] Nov 01 22:43:46.085138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:46.080662 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7021-aa3d9b" Nov 01 22:43:46.090182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:46.089595 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vrfk9\"" Nov 01 22:43:46.156950 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:46.156912 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjv96\" (UniqueName: \"kubernetes.io/projected/8863c51b-95bf-4ae3-95c6-ebab82155497-kube-api-access-sjv96\") pod \"test-curl-7021-aa3d9b\" (UID: \"8863c51b-95bf-4ae3-95c6-ebab82155497\") " pod="opendatahub/test-curl-7021-aa3d9b" Nov 01 22:43:46.258847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:46.258814 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjv96\" (UniqueName: \"kubernetes.io/projected/8863c51b-95bf-4ae3-95c6-ebab82155497-kube-api-access-sjv96\") pod \"test-curl-7021-aa3d9b\" (UID: \"8863c51b-95bf-4ae3-95c6-ebab82155497\") " pod="opendatahub/test-curl-7021-aa3d9b" Nov 01 22:43:46.267324 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:46.267293 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjv96\" (UniqueName: \"kubernetes.io/projected/8863c51b-95bf-4ae3-95c6-ebab82155497-kube-api-access-sjv96\") pod \"test-curl-7021-aa3d9b\" (UID: \"8863c51b-95bf-4ae3-95c6-ebab82155497\") " pod="opendatahub/test-curl-7021-aa3d9b" Nov 01 22:43:46.418139 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:46.412419 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7021-aa3d9b" Nov 01 22:43:46.916397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:46.916366 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7021-aa3d9b"] Nov 01 22:43:46.916530 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:43:46.916425 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8863c51b_95bf_4ae3_95c6_ebab82155497.slice/crio-2fe3770b039de64b7ad09d2952cd0cbb32a508f53da08b97e9218d3008282515 WatchSource:0}: Error finding container 2fe3770b039de64b7ad09d2952cd0cbb32a508f53da08b97e9218d3008282515: Status 404 returned error can't find the container with id 2fe3770b039de64b7ad09d2952cd0cbb32a508f53da08b97e9218d3008282515 Nov 01 22:43:47.779041 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:47.775122 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7021-aa3d9b" event={"ID":"8863c51b-95bf-4ae3-95c6-ebab82155497","Type":"ContainerStarted","Data":"002e3643020ac9508bd2b51df2fb4bbd5ca15fb4586a7c59fd108477a0ec2081"} Nov 01 22:43:47.779041 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:47.775187 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7021-aa3d9b" event={"ID":"8863c51b-95bf-4ae3-95c6-ebab82155497","Type":"ContainerStarted","Data":"2fe3770b039de64b7ad09d2952cd0cbb32a508f53da08b97e9218d3008282515"} Nov 01 22:43:49.296010 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:43:49.295964 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:43:49.323015 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:49.322962 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7021-aa3d9b" podStartSLOduration=4.322945144 podStartE2EDuration="4.322945144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:47.814482142 +0000 UTC m=+1231.598921060" watchObservedRunningTime="2026-11-01 22:43:49.322945144 +0000 UTC m=+1233.107384064" Nov 01 22:43:49.336156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:49.330497 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7021-aa3d9b"] Nov 01 22:43:49.343147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:49.338809 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7021-aa3d9b"] Nov 01 22:43:49.875654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:49.875609 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7021-aa3d9b" podUID="8863c51b-95bf-4ae3-95c6-ebab82155497" containerName="test-curl-7021-aa3d9b" containerID="cri-o://002e3643020ac9508bd2b51df2fb4bbd5ca15fb4586a7c59fd108477a0ec2081" gracePeriod=2 Nov 01 22:43:49.878341 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:43:49.878299 6466 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7021-aa3d9b\" is forbidden: User \"system:node:ip-10-0-134-88.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-88.us-east-2.compute.internal' and this object" podUID="8863c51b-95bf-4ae3-95c6-ebab82155497" pod="opendatahub/test-curl-7021-aa3d9b" Nov 01 22:43:50.256591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:50.256549 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh"] Nov 01 22:43:50.286682 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:43:50.286632 6466 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7021-aa3d9b\" is forbidden: User \"system:node:ip-10-0-134-88.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-88.us-east-2.compute.internal' and this object" podUID="8863c51b-95bf-4ae3-95c6-ebab82155497" pod="opendatahub/test-curl-7021-aa3d9b" Nov 01 22:43:51.963402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:51.963368 6466 generic.go:356] "Generic (PLEG): container finished" podID="8863c51b-95bf-4ae3-95c6-ebab82155497" containerID="002e3643020ac9508bd2b51df2fb4bbd5ca15fb4586a7c59fd108477a0ec2081" exitCode=137 Nov 01 22:43:52.116323 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:52.116294 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7021-aa3d9b" Nov 01 22:43:52.208237 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:52.208203 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8863c51b-95bf-4ae3-95c6-ebab82155497-kube-api-access-sjv96\" (UniqueName: \"kubernetes.io/projected/8863c51b-95bf-4ae3-95c6-ebab82155497-kube-api-access-sjv96\") pod \"8863c51b-95bf-4ae3-95c6-ebab82155497\" (UID: \"8863c51b-95bf-4ae3-95c6-ebab82155497\") " Nov 01 22:43:52.210587 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:52.210552 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8863c51b-95bf-4ae3-95c6-ebab82155497-kube-api-access-sjv96" pod "8863c51b-95bf-4ae3-95c6-ebab82155497" (UID: "8863c51b-95bf-4ae3-95c6-ebab82155497"). InnerVolumeSpecName "kube-api-access-sjv96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:52.311561 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:52.311522 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjv96\" (UniqueName: \"kubernetes.io/projected/8863c51b-95bf-4ae3-95c6-ebab82155497-kube-api-access-sjv96\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:52.346444 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:52.346413 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8863c51b-95bf-4ae3-95c6-ebab82155497" path="/var/lib/kubelet/pods/8863c51b-95bf-4ae3-95c6-ebab82155497/volumes" Nov 01 22:43:52.973435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:52.973408 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7021-aa3d9b" Nov 01 22:43:52.973895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:52.973467 6466 scope.go:122] "RemoveContainer" containerID="002e3643020ac9508bd2b51df2fb4bbd5ca15fb4586a7c59fd108477a0ec2081" Nov 01 22:43:55.257154 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:55.257114 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" podUID="556faa39-b98e-4133-adea-bce7ed14751a" containerName="payload-processing" containerID="cri-o://2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338" gracePeriod=25 Nov 01 22:43:55.656975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:55.656945 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:43:55.729362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:55.729331 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556faa39-b98e-4133-adea-bce7ed14751a-kube-api-access-g757s\" (UniqueName: \"kubernetes.io/projected/556faa39-b98e-4133-adea-bce7ed14751a-kube-api-access-g757s\") pod \"556faa39-b98e-4133-adea-bce7ed14751a\" (UID: \"556faa39-b98e-4133-adea-bce7ed14751a\") " Nov 01 22:43:55.729557 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:55.729543 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/556faa39-b98e-4133-adea-bce7ed14751a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556faa39-b98e-4133-adea-bce7ed14751a-plugins-config-volume\") pod \"556faa39-b98e-4133-adea-bce7ed14751a\" (UID: \"556faa39-b98e-4133-adea-bce7ed14751a\") " Nov 01 22:43:55.729907 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:55.729883 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556faa39-b98e-4133-adea-bce7ed14751a-plugins-config-volume" pod "556faa39-b98e-4133-adea-bce7ed14751a" (UID: "556faa39-b98e-4133-adea-bce7ed14751a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:55.730244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:55.730230 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556faa39-b98e-4133-adea-bce7ed14751a-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:55.731758 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:55.731735 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556faa39-b98e-4133-adea-bce7ed14751a-kube-api-access-g757s" pod "556faa39-b98e-4133-adea-bce7ed14751a" (UID: "556faa39-b98e-4133-adea-bce7ed14751a"). InnerVolumeSpecName "kube-api-access-g757s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:55.832102 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:55.832067 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g757s\" (UniqueName: \"kubernetes.io/projected/556faa39-b98e-4133-adea-bce7ed14751a-kube-api-access-g757s\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:56.005477 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:56.005443 6466 generic.go:356] "Generic (PLEG): container finished" podID="556faa39-b98e-4133-adea-bce7ed14751a" containerID="2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338" exitCode=0 Nov 01 22:43:56.005690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:56.005502 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" Nov 01 22:43:56.005690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:56.005527 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" event={"ID":"556faa39-b98e-4133-adea-bce7ed14751a","Type":"ContainerDied","Data":"2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338"} Nov 01 22:43:56.005690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:56.005587 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh" event={"ID":"556faa39-b98e-4133-adea-bce7ed14751a","Type":"ContainerDied","Data":"f7440076c8a24f150127a3768e7cc24261447d75a0a763dd23c95c2afbe5b4c4"} Nov 01 22:43:56.005690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:56.005615 6466 scope.go:122] "RemoveContainer" containerID="2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338" Nov 01 22:43:56.023438 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:56.023415 6466 scope.go:122] "RemoveContainer" containerID="2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338" Nov 01 22:43:56.023768 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:43:56.023736 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338\": container with ID starting with 2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338 not found: ID does not exist" containerID="2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338" Nov 01 22:43:56.023880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:56.023775 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338"} err="failed to get container status \"2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338\": rpc error: code = NotFound desc = could not find container \"2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338\": container with ID starting with 2b483997430eaae95f55d79afa529790cf1b353a7499cb09f830b29f0a1c4338 not found: ID does not exist" Nov 01 22:43:56.033120 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:56.033085 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh"] Nov 01 22:43:56.039051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:56.039030 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee8d1a-79fb7d8cd7-zslmh"] Nov 01 22:43:56.344180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:56.344093 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556faa39-b98e-4133-adea-bce7ed14751a" path="/var/lib/kubelet/pods/556faa39-b98e-4133-adea-bce7ed14751a/volumes" Nov 01 22:43:59.703695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:59.703652 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct"] Nov 01 22:43:59.711785 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:43:59.711757 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6"] Nov 01 22:44:04.704771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:04.704717 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" podUID="49592222-eba1-4751-98e0-0aaba69e8167" containerName="payload-processing" containerID="cri-o://7ccdc531ecaede15c2eec9d80ce44a316e5c99b9f6324524db82062cf164783e" gracePeriod=25 Nov 01 22:44:04.712469 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:04.712411 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" podUID="707f0b16-452d-4adc-ac7d-5e3cb927618f" containerName="payload-pre-processing" containerID="cri-o://ca7f7b62465aa77a1654fdae3214ef379321eeb71aedb5f4b5aaf83e47581ccd" gracePeriod=25 Nov 01 22:44:05.412157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.410204 6466 generic.go:356] "Generic (PLEG): container finished" podID="49592222-eba1-4751-98e0-0aaba69e8167" containerID="7ccdc531ecaede15c2eec9d80ce44a316e5c99b9f6324524db82062cf164783e" exitCode=0 Nov 01 22:44:05.412157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.410457 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" event={"ID":"49592222-eba1-4751-98e0-0aaba69e8167","Type":"ContainerDied","Data":"7ccdc531ecaede15c2eec9d80ce44a316e5c99b9f6324524db82062cf164783e"} Nov 01 22:44:05.412869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.412711 6466 generic.go:356] "Generic (PLEG): container finished" podID="707f0b16-452d-4adc-ac7d-5e3cb927618f" containerID="ca7f7b62465aa77a1654fdae3214ef379321eeb71aedb5f4b5aaf83e47581ccd" exitCode=0 Nov 01 22:44:05.412869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.412831 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" event={"ID":"707f0b16-452d-4adc-ac7d-5e3cb927618f","Type":"ContainerDied","Data":"ca7f7b62465aa77a1654fdae3214ef379321eeb71aedb5f4b5aaf83e47581ccd"} Nov 01 22:44:05.458760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.458709 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:44:05.509847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.509811 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49592222-eba1-4751-98e0-0aaba69e8167-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49592222-eba1-4751-98e0-0aaba69e8167-plugins-config-volume\") pod \"49592222-eba1-4751-98e0-0aaba69e8167\" (UID: \"49592222-eba1-4751-98e0-0aaba69e8167\") " Nov 01 22:44:05.510055 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.510039 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49592222-eba1-4751-98e0-0aaba69e8167-kube-api-access-zhchq\" (UniqueName: \"kubernetes.io/projected/49592222-eba1-4751-98e0-0aaba69e8167-kube-api-access-zhchq\") pod \"49592222-eba1-4751-98e0-0aaba69e8167\" (UID: \"49592222-eba1-4751-98e0-0aaba69e8167\") " Nov 01 22:44:05.510317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.510285 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49592222-eba1-4751-98e0-0aaba69e8167-plugins-config-volume" pod "49592222-eba1-4751-98e0-0aaba69e8167" (UID: "49592222-eba1-4751-98e0-0aaba69e8167"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:05.510973 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.510947 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49592222-eba1-4751-98e0-0aaba69e8167-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:05.512469 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.512442 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49592222-eba1-4751-98e0-0aaba69e8167-kube-api-access-zhchq" pod "49592222-eba1-4751-98e0-0aaba69e8167" (UID: "49592222-eba1-4751-98e0-0aaba69e8167"). InnerVolumeSpecName "kube-api-access-zhchq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:05.539337 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.539309 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:44:05.626264 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.623795 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707f0b16-452d-4adc-ac7d-5e3cb927618f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707f0b16-452d-4adc-ac7d-5e3cb927618f-plugins-config-volume\") pod \"707f0b16-452d-4adc-ac7d-5e3cb927618f\" (UID: \"707f0b16-452d-4adc-ac7d-5e3cb927618f\") " Nov 01 22:44:05.626264 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.624221 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707f0b16-452d-4adc-ac7d-5e3cb927618f-kube-api-access-8k57t\" (UniqueName: \"kubernetes.io/projected/707f0b16-452d-4adc-ac7d-5e3cb927618f-kube-api-access-8k57t\") pod \"707f0b16-452d-4adc-ac7d-5e3cb927618f\" (UID: \"707f0b16-452d-4adc-ac7d-5e3cb927618f\") " Nov 01 22:44:05.680191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.626818 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707f0b16-452d-4adc-ac7d-5e3cb927618f-plugins-config-volume" pod "707f0b16-452d-4adc-ac7d-5e3cb927618f" (UID: "707f0b16-452d-4adc-ac7d-5e3cb927618f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:05.680191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.630218 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707f0b16-452d-4adc-ac7d-5e3cb927618f-kube-api-access-8k57t" pod "707f0b16-452d-4adc-ac7d-5e3cb927618f" (UID: "707f0b16-452d-4adc-ac7d-5e3cb927618f"). InnerVolumeSpecName "kube-api-access-8k57t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:05.680191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.636635 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k57t\" (UniqueName: \"kubernetes.io/projected/707f0b16-452d-4adc-ac7d-5e3cb927618f-kube-api-access-8k57t\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:05.680191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.636659 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhchq\" (UniqueName: \"kubernetes.io/projected/49592222-eba1-4751-98e0-0aaba69e8167-kube-api-access-zhchq\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:05.680191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:05.636673 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/707f0b16-452d-4adc-ac7d-5e3cb927618f-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:06.507626 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:06.507598 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" Nov 01 22:44:06.513039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:06.512740 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct" event={"ID":"49592222-eba1-4751-98e0-0aaba69e8167","Type":"ContainerDied","Data":"fd6627583e45332a41b7e73e194a9d2167c2174c0f7e6b7c308d7b65738a1972"} Nov 01 22:44:06.513465 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:06.513440 6466 scope.go:122] "RemoveContainer" containerID="7ccdc531ecaede15c2eec9d80ce44a316e5c99b9f6324524db82062cf164783e" Nov 01 22:44:06.522461 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:06.522428 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" event={"ID":"707f0b16-452d-4adc-ac7d-5e3cb927618f","Type":"ContainerDied","Data":"029820a739159abd7da003b6fc89c0f743e91fba363dcd3001fbd867c10a0eb6"} Nov 01 22:44:06.539064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:06.535838 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6" Nov 01 22:44:06.614133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:06.605185 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6"] Nov 01 22:44:06.634234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:06.616458 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-t94g6"] Nov 01 22:44:06.705142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:06.696770 6466 scope.go:122] "RemoveContainer" containerID="ca7f7b62465aa77a1654fdae3214ef379321eeb71aedb5f4b5aaf83e47581ccd" Nov 01 22:44:06.720454 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:06.720416 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct"] Nov 01 22:44:06.720733 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:06.720715 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-425870c1-5d9b47cd7f-w6bct"] Nov 01 22:44:08.343282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:08.343253 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49592222-eba1-4751-98e0-0aaba69e8167" path="/var/lib/kubelet/pods/49592222-eba1-4751-98e0-0aaba69e8167/volumes" Nov 01 22:44:08.343768 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:08.343754 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707f0b16-452d-4adc-ac7d-5e3cb927618f" path="/var/lib/kubelet/pods/707f0b16-452d-4adc-ac7d-5e3cb927618f/volumes" Nov 01 22:44:11.839268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.839232 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv"] Nov 01 22:44:11.840158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840137 6466 cpu_manager.go:414] "containerMap: removing container" podUID="556faa39-b98e-4133-adea-bce7ed14751a" containerName="payload-processing" Nov 01 22:44:11.840158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840155 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556faa39-b98e-4133-adea-bce7ed14751a" containerName="payload-processing" Nov 01 22:44:11.840347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840171 6466 cpu_manager.go:414] "containerMap: removing container" podUID="707f0b16-452d-4adc-ac7d-5e3cb927618f" containerName="payload-pre-processing" Nov 01 22:44:11.840347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840177 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707f0b16-452d-4adc-ac7d-5e3cb927618f" containerName="payload-pre-processing" Nov 01 22:44:11.840347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840217 6466 cpu_manager.go:414] "containerMap: removing container" podUID="8863c51b-95bf-4ae3-95c6-ebab82155497" containerName="test-curl-7021-aa3d9b" Nov 01 22:44:11.840347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840222 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8863c51b-95bf-4ae3-95c6-ebab82155497" containerName="test-curl-7021-aa3d9b" Nov 01 22:44:11.840347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840241 6466 cpu_manager.go:414] "containerMap: removing container" podUID="49592222-eba1-4751-98e0-0aaba69e8167" containerName="payload-processing" Nov 01 22:44:11.840347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840246 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49592222-eba1-4751-98e0-0aaba69e8167" containerName="payload-processing" Nov 01 22:44:11.840628 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840367 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="8863c51b-95bf-4ae3-95c6-ebab82155497" containerName="test-curl-7021-aa3d9b" Nov 01 22:44:11.840628 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840378 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="49592222-eba1-4751-98e0-0aaba69e8167" containerName="payload-processing" Nov 01 22:44:11.840628 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840387 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="556faa39-b98e-4133-adea-bce7ed14751a" containerName="payload-processing" Nov 01 22:44:11.840628 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.840396 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="707f0b16-452d-4adc-ac7d-5e3cb927618f" containerName="payload-pre-processing" Nov 01 22:44:11.856039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.854836 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:11.865886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.865860 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ae0e3a-openshift-default-dockercfg-xvj5h\"" Nov 01 22:44:11.875030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.874993 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv"] Nov 01 22:44:11.936822 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.936788 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llccn\" (UniqueName: \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-kube-api-access-llccn\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:11.937003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.936904 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-socket\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:11.937143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.937094 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-credential-socket\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:11.937272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.937243 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-data\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:11.937419 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.937378 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-certs\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:11.937486 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.937451 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43636efd-d17f-4e43-bb3d-f82b63999337-istiod-ca-cert\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:11.937554 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.937539 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-envoy\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:11.937610 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.937600 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43636efd-d17f-4e43-bb3d-f82b63999337-istio-podinfo\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:11.937682 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:11.937642 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-istio-token\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.038905 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.038867 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-credential-socket\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039125 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.038971 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-data\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039125 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039046 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-certs\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039125 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039080 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43636efd-d17f-4e43-bb3d-f82b63999337-istiod-ca-cert\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039310 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039209 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-envoy\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039310 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039248 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43636efd-d17f-4e43-bb3d-f82b63999337-istio-podinfo\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039310 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039283 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-credential-socket\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039379 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-data\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039387 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-istio-token\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039428 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-certs\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039444 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llccn\" (UniqueName: \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-kube-api-access-llccn\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039517 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-socket\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.039898 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039868 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43636efd-d17f-4e43-bb3d-f82b63999337-istiod-ca-cert\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.040004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.039883 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-socket\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.041726 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.041703 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-envoy\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.041955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.041935 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43636efd-d17f-4e43-bb3d-f82b63999337-istio-podinfo\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.049133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.049097 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-istio-token\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.049933 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.049910 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llccn\" (UniqueName: \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-kube-api-access-llccn\") pod \"e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:12.184155 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:12.182466 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:13.016208 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:13.016181 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv"] Nov 01 22:44:13.018755 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:44:13.018728 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43636efd_d17f_4e43_bb3d_f82b63999337.slice/crio-d56aae6232c2009991fccd4b10aa76fbe1f1662a430766444c4fcc8e92667616 WatchSource:0}: Error finding container d56aae6232c2009991fccd4b10aa76fbe1f1662a430766444c4fcc8e92667616: Status 404 returned error can't find the container with id d56aae6232c2009991fccd4b10aa76fbe1f1662a430766444c4fcc8e92667616 Nov 01 22:44:13.022659 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:13.022625 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:44:13.022747 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:13.022698 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:44:13.022789 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:13.022762 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:44:13.962271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:13.962233 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" event={"ID":"43636efd-d17f-4e43-bb3d-f82b63999337","Type":"ContainerStarted","Data":"7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c"} Nov 01 22:44:13.962506 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:13.962485 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" event={"ID":"43636efd-d17f-4e43-bb3d-f82b63999337","Type":"ContainerStarted","Data":"d56aae6232c2009991fccd4b10aa76fbe1f1662a430766444c4fcc8e92667616"} Nov 01 22:44:14.041131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:14.034502 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" podStartSLOduration=3.034478528 podStartE2EDuration="3.034478528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:14.028462437 +0000 UTC m=+1257.812901361" watchObservedRunningTime="2026-11-01 22:44:14.034478528 +0000 UTC m=+1257.818917451" Nov 01 22:44:14.183016 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:14.182972 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:14.256221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:14.256140 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:14.979193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:14.978546 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:14.984204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:14.980551 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:44:17.666578 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:17.666542 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz"] Nov 01 22:44:17.673153 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:17.673124 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:44:17.688320 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:17.688288 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ae0e3a\"" Nov 01 22:44:17.689203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:17.689175 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz"] Nov 01 22:44:17.756590 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:17.756557 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22946ea8-86f3-4945-93a7-7e100bcf37a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz\" (UID: \"22946ea8-86f3-4945-93a7-7e100bcf37a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:44:17.756795 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:17.756629 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc7m8\" (UniqueName: \"kubernetes.io/projected/22946ea8-86f3-4945-93a7-7e100bcf37a9-kube-api-access-mc7m8\") pod \"payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz\" (UID: \"22946ea8-86f3-4945-93a7-7e100bcf37a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:44:17.904129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:17.878563 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22946ea8-86f3-4945-93a7-7e100bcf37a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz\" (UID: \"22946ea8-86f3-4945-93a7-7e100bcf37a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:44:17.904499 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:17.904467 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc7m8\" (UniqueName: \"kubernetes.io/projected/22946ea8-86f3-4945-93a7-7e100bcf37a9-kube-api-access-mc7m8\") pod \"payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz\" (UID: \"22946ea8-86f3-4945-93a7-7e100bcf37a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:44:17.905521 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:17.893123 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22946ea8-86f3-4945-93a7-7e100bcf37a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz\" (UID: \"22946ea8-86f3-4945-93a7-7e100bcf37a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:44:17.941007 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:17.940922 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc7m8\" (UniqueName: \"kubernetes.io/projected/22946ea8-86f3-4945-93a7-7e100bcf37a9-kube-api-access-mc7m8\") pod \"payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz\" (UID: \"22946ea8-86f3-4945-93a7-7e100bcf37a9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:44:17.998134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:17.994454 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:44:18.557250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:18.557215 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz"] Nov 01 22:44:18.557899 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:44:18.557872 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22946ea8_86f3_4945_93a7_7e100bcf37a9.slice/crio-382ff7d7ec4510243a6a6bd859354586da208d515ca3bcda6127c9b57a978223 WatchSource:0}: Error finding container 382ff7d7ec4510243a6a6bd859354586da208d515ca3bcda6127c9b57a978223: Status 404 returned error can't find the container with id 382ff7d7ec4510243a6a6bd859354586da208d515ca3bcda6127c9b57a978223 Nov 01 22:44:19.331161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:19.317020 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" event={"ID":"22946ea8-86f3-4945-93a7-7e100bcf37a9","Type":"ContainerStarted","Data":"2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5"} Nov 01 22:44:19.331161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:19.317096 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" event={"ID":"22946ea8-86f3-4945-93a7-7e100bcf37a9","Type":"ContainerStarted","Data":"382ff7d7ec4510243a6a6bd859354586da208d515ca3bcda6127c9b57a978223"} Nov 01 22:44:19.331161 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:19.318532 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:44:19.412464 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:44:19.412428 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:44:20.375388 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.375302 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" podStartSLOduration=3.375283421 podStartE2EDuration="3.375283421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:19.366496607 +0000 UTC m=+1263.150935537" watchObservedRunningTime="2026-11-01 22:44:20.375283421 +0000 UTC m=+1264.159722343" Nov 01 22:44:20.389429 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.389396 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x"] Nov 01 22:44:20.389602 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.389584 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" containerID="cri-o://4edc1e58707ec0d31e717b79207875d54725dd8d967879d007689fe353db237d" gracePeriod=30 Nov 01 22:44:20.391967 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.391940 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7"] Nov 01 22:44:20.392166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.392140 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" podUID="27c48728-a545-4f28-a74a-99f701418e9d" containerName="istio-proxy" containerID="cri-o://de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a" gracePeriod=30 Nov 01 22:44:20.408695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.399216 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq"] Nov 01 22:44:20.408695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.399407 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" containerID="cri-o://5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb" gracePeriod=30 Nov 01 22:44:20.408695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.406370 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg"] Nov 01 22:44:20.408695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.406576 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" containerID="cri-o://a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8" gracePeriod=30 Nov 01 22:44:20.415285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.412727 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv"] Nov 01 22:44:20.415285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.412914 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" containerID="cri-o://dc53a77f9fc6526d7974b64406175f2742531b105f6805db539b2ab96c174ce9" gracePeriod=30 Nov 01 22:44:20.426140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.423704 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz"] Nov 01 22:44:20.426140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:20.423905 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" containerID="cri-o://ede69a7260c2703129e4749bc566c595b8fc73843a03869a4bb1dd7256224d1b" gracePeriod=30 Nov 01 22:44:25.105791 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:25.105755 6466 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:44:25.106221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:25.105814 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:44:26.284500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.283966 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:44:26.298771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.298741 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5"] Nov 01 22:44:26.420420 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.420380 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-data\") pod \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " Nov 01 22:44:26.420745 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.420718 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-data" pod "84e4b70b-8d5e-4d95-9a62-3996d9fac057" (UID: "84e4b70b-8d5e-4d95-9a62-3996d9fac057"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.421048 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.421027 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-envoy\") pod \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " Nov 01 22:44:26.421305 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.421286 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-socket\") pod \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " Nov 01 22:44:26.421483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.421466 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-credential-socket\") pod \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " Nov 01 22:44:26.421617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.421602 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-token\" (UniqueName: \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-token\") pod \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " Nov 01 22:44:26.421753 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.421740 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-certs\") pod \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " Nov 01 22:44:26.421882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.421869 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-podinfo\") pod \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " Nov 01 22:44:26.422022 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.422009 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-kube-api-access-9nmkc\" (UniqueName: \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-kube-api-access-9nmkc\") pod \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " Nov 01 22:44:26.422196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.422179 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istiod-ca-cert\") pod \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\" (UID: \"84e4b70b-8d5e-4d95-9a62-3996d9fac057\") " Nov 01 22:44:26.422366 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.422342 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-credential-socket" pod "84e4b70b-8d5e-4d95-9a62-3996d9fac057" (UID: "84e4b70b-8d5e-4d95-9a62-3996d9fac057"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.423214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.422775 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-socket" pod "84e4b70b-8d5e-4d95-9a62-3996d9fac057" (UID: "84e4b70b-8d5e-4d95-9a62-3996d9fac057"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.423214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.422952 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-certs" pod "84e4b70b-8d5e-4d95-9a62-3996d9fac057" (UID: "84e4b70b-8d5e-4d95-9a62-3996d9fac057"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.424034 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.424010 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.424327 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.424312 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.424461 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.424447 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.424565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.424553 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.425037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.425009 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istiod-ca-cert" pod "84e4b70b-8d5e-4d95-9a62-3996d9fac057" (UID: "84e4b70b-8d5e-4d95-9a62-3996d9fac057"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:26.427634 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.427615 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:44:26.432011 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.431961 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:44:26.432213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.432024 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-podinfo" pod "84e4b70b-8d5e-4d95-9a62-3996d9fac057" (UID: "84e4b70b-8d5e-4d95-9a62-3996d9fac057"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:26.433561 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.433540 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-kube-api-access-9nmkc" pod "84e4b70b-8d5e-4d95-9a62-3996d9fac057" (UID: "84e4b70b-8d5e-4d95-9a62-3996d9fac057"). InnerVolumeSpecName "kube-api-access-9nmkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:26.433676 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.433579 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-token" pod "84e4b70b-8d5e-4d95-9a62-3996d9fac057" (UID: "84e4b70b-8d5e-4d95-9a62-3996d9fac057"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:26.433897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.433878 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-envoy" pod "84e4b70b-8d5e-4d95-9a62-3996d9fac057" (UID: "84e4b70b-8d5e-4d95-9a62-3996d9fac057"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.466854 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.466818 6466 generic.go:356] "Generic (PLEG): container finished" podID="330ac018-be0b-4502-9562-b0999cdc8245" containerID="4edc1e58707ec0d31e717b79207875d54725dd8d967879d007689fe353db237d" exitCode=0 Nov 01 22:44:26.469931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.469900 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" Nov 01 22:44:26.474271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.469730 6466 generic.go:356] "Generic (PLEG): container finished" podID="27c48728-a545-4f28-a74a-99f701418e9d" containerID="de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a" exitCode=0 Nov 01 22:44:26.474271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.473060 6466 generic.go:356] "Generic (PLEG): container finished" podID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerID="ede69a7260c2703129e4749bc566c595b8fc73843a03869a4bb1dd7256224d1b" exitCode=0 Nov 01 22:44:26.475754 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.475379 6466 generic.go:356] "Generic (PLEG): container finished" podID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerID="5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb" exitCode=0 Nov 01 22:44:26.476160 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.476059 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" Nov 01 22:44:26.480941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.480624 6466 generic.go:356] "Generic (PLEG): container finished" podID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerID="a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8" exitCode=0 Nov 01 22:44:26.480941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.480768 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" Nov 01 22:44:26.490519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.490493 6466 generic.go:356] "Generic (PLEG): container finished" podID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerID="dc53a77f9fc6526d7974b64406175f2742531b105f6805db539b2ab96c174ce9" exitCode=0 Nov 01 22:44:26.529049 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.529017 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.529376 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.529052 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.529376 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.529069 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.529376 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.529084 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nmkc\" (UniqueName: \"kubernetes.io/projected/84e4b70b-8d5e-4d95-9a62-3996d9fac057-kube-api-access-9nmkc\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.529376 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.529099 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e4b70b-8d5e-4d95-9a62-3996d9fac057-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.538381 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.538364 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8"] Nov 01 22:44:26.538561 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.538543 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" event={"ID":"330ac018-be0b-4502-9562-b0999cdc8245","Type":"ContainerDied","Data":"4edc1e58707ec0d31e717b79207875d54725dd8d967879d007689fe353db237d"} Nov 01 22:44:26.538649 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.538638 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" event={"ID":"27c48728-a545-4f28-a74a-99f701418e9d","Type":"ContainerDied","Data":"de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a"} Nov 01 22:44:26.538721 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.538711 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7" event={"ID":"27c48728-a545-4f28-a74a-99f701418e9d","Type":"ContainerDied","Data":"b7ad3f6375496afc8c06487b0a80b8518d0a017cef0a6fc3c8b811689329863c"} Nov 01 22:44:26.538807 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.538794 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" event={"ID":"3dbf3aa0-4a0b-4069-b219-6f73873bf995","Type":"ContainerDied","Data":"ede69a7260c2703129e4749bc566c595b8fc73843a03869a4bb1dd7256224d1b"} Nov 01 22:44:26.538883 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.538874 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" event={"ID":"eb948b12-1e45-4e5f-9750-eed6b5ad09cf","Type":"ContainerDied","Data":"5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb"} Nov 01 22:44:26.538946 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.538938 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq" event={"ID":"eb948b12-1e45-4e5f-9750-eed6b5ad09cf","Type":"ContainerDied","Data":"07579e79bae96233f83b469f08a79004cdc8d695718576039fa45d591240c347"} Nov 01 22:44:26.539011 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.539002 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" event={"ID":"84e4b70b-8d5e-4d95-9a62-3996d9fac057","Type":"ContainerDied","Data":"a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8"} Nov 01 22:44:26.539186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.539168 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg" event={"ID":"84e4b70b-8d5e-4d95-9a62-3996d9fac057","Type":"ContainerDied","Data":"df278f54efb9f68d9c05a2a22c2c8db9a1aad2c293c0e9883d71abb02c72d12e"} Nov 01 22:44:26.539294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.539278 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" event={"ID":"6691f77d-0336-4b20-a16f-9d8b5f2159e2","Type":"ContainerDied","Data":"dc53a77f9fc6526d7974b64406175f2742531b105f6805db539b2ab96c174ce9"} Nov 01 22:44:26.539366 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.539020 6466 scope.go:122] "RemoveContainer" containerID="de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a" Nov 01 22:44:26.578129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.567456 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.630341 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-data\") pod \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.630393 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-credential-socket\") pod \"27c48728-a545-4f28-a74a-99f701418e9d\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.630437 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istiod-ca-cert\") pod \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.630537 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-kube-api-access-5wmhg\" (UniqueName: \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-kube-api-access-5wmhg\") pod \"27c48728-a545-4f28-a74a-99f701418e9d\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.630580 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-podinfo\") pod \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.630624 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-istio-token\" (UniqueName: \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-istio-token\") pod \"27c48728-a545-4f28-a74a-99f701418e9d\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.630655 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-certs\") pod \"27c48728-a545-4f28-a74a-99f701418e9d\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.630711 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-kube-api-access-6wmrm\" (UniqueName: \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-kube-api-access-6wmrm\") pod \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.630931 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-token\" (UniqueName: \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-token\") pod \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631008 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-envoy\") pod \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631051 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-certs\") pod \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631089 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-credential-socket\") pod \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631159 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-envoy\") pod \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631192 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-podinfo\") pod \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631242 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-data\") pod \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631271 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27c48728-a545-4f28-a74a-99f701418e9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c48728-a545-4f28-a74a-99f701418e9d-istio-podinfo\") pod \"27c48728-a545-4f28-a74a-99f701418e9d\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631316 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-socket\") pod \"27c48728-a545-4f28-a74a-99f701418e9d\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631450 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-data\") pod \"27c48728-a545-4f28-a74a-99f701418e9d\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631478 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c48728-a545-4f28-a74a-99f701418e9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c48728-a545-4f28-a74a-99f701418e9d-istiod-ca-cert\") pod \"27c48728-a545-4f28-a74a-99f701418e9d\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631508 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-socket\") pod \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631619 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-credential-socket\") pod \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\" (UID: \"eb948b12-1e45-4e5f-9750-eed6b5ad09cf\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631659 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-envoy\") pod \"27c48728-a545-4f28-a74a-99f701418e9d\" (UID: \"27c48728-a545-4f28-a74a-99f701418e9d\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.631799 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-certs\") pod \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " Nov 01 22:44:26.654541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.649297 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-certs" pod "3dbf3aa0-4a0b-4069-b219-6f73873bf995" (UID: "3dbf3aa0-4a0b-4069-b219-6f73873bf995"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.656382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.656056 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-certs" pod "27c48728-a545-4f28-a74a-99f701418e9d" (UID: "27c48728-a545-4f28-a74a-99f701418e9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.679191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.668393 6466 scope.go:122] "RemoveContainer" containerID="de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.700352 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:44:26.701005 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a\": container with ID starting with de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a not found: ID does not exist" containerID="de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.701048 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a"} err="failed to get container status \"de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a\": rpc error: code = NotFound desc = could not find container \"de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a\": container with ID starting with de91971deedbf8d9aff4cc8d91807f72d4e2514b8bde88d037b2b4027807ef9a not found: ID does not exist" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.701077 6466 scope.go:122] "RemoveContainer" containerID="5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.701571 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c48728-a545-4f28-a74a-99f701418e9d-istiod-ca-cert" pod "27c48728-a545-4f28-a74a-99f701418e9d" (UID: "27c48728-a545-4f28-a74a-99f701418e9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.701820 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-socket" pod "27c48728-a545-4f28-a74a-99f701418e9d" (UID: "27c48728-a545-4f28-a74a-99f701418e9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.701988 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-data" pod "27c48728-a545-4f28-a74a-99f701418e9d" (UID: "27c48728-a545-4f28-a74a-99f701418e9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.702170 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-credential-socket" pod "27c48728-a545-4f28-a74a-99f701418e9d" (UID: "27c48728-a545-4f28-a74a-99f701418e9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.702398 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-data" pod "eb948b12-1e45-4e5f-9750-eed6b5ad09cf" (UID: "eb948b12-1e45-4e5f-9750-eed6b5ad09cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.702596 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-credential-socket" pod "eb948b12-1e45-4e5f-9750-eed6b5ad09cf" (UID: "eb948b12-1e45-4e5f-9750-eed6b5ad09cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.702801 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-socket" pod "eb948b12-1e45-4e5f-9750-eed6b5ad09cf" (UID: "eb948b12-1e45-4e5f-9750-eed6b5ad09cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.718234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.703101 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istiod-ca-cert" pod "eb948b12-1e45-4e5f-9750-eed6b5ad09cf" (UID: "eb948b12-1e45-4e5f-9750-eed6b5ad09cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:26.743291 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.742897 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-credential-socket" pod "3dbf3aa0-4a0b-4069-b219-6f73873bf995" (UID: "3dbf3aa0-4a0b-4069-b219-6f73873bf995"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.793330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.792931 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg"] Nov 01 22:44:26.793510 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.793364 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-certs" pod "eb948b12-1e45-4e5f-9750-eed6b5ad09cf" (UID: "eb948b12-1e45-4e5f-9750-eed6b5ad09cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.793713 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.794718 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-data" pod "3dbf3aa0-4a0b-4069-b219-6f73873bf995" (UID: "3dbf3aa0-4a0b-4069-b219-6f73873bf995"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.796185 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-token\" (UniqueName: \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-token\") pod \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.796500 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-credential-socket\") pod \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.796555 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330ac018-be0b-4502-9562-b0999cdc8245-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330ac018-be0b-4502-9562-b0999cdc8245-istiod-ca-cert\") pod \"330ac018-be0b-4502-9562-b0999cdc8245\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.796660 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-envoy\") pod \"330ac018-be0b-4502-9562-b0999cdc8245\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.796722 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istiod-ca-cert\") pod \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.796862 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-socket\") pod \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.796929 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-data\") pod \"330ac018-be0b-4502-9562-b0999cdc8245\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.796979 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-podinfo\") pod \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797076 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-credential-socket\") pod \"330ac018-be0b-4502-9562-b0999cdc8245\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797193 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-data\") pod \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797299 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-kube-api-access-r4h8q\" (UniqueName: \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-kube-api-access-r4h8q\") pod \"330ac018-be0b-4502-9562-b0999cdc8245\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797363 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-istio-token\" (UniqueName: \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-istio-token\") pod \"330ac018-be0b-4502-9562-b0999cdc8245\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797443 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/330ac018-be0b-4502-9562-b0999cdc8245-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330ac018-be0b-4502-9562-b0999cdc8245-istio-podinfo\") pod \"330ac018-be0b-4502-9562-b0999cdc8245\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797556 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-socket\") pod \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797612 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-kube-api-access-rsfsv\" (UniqueName: \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-kube-api-access-rsfsv\") pod \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797672 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-certs\") pod \"330ac018-be0b-4502-9562-b0999cdc8245\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797718 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-socket\") pod \"330ac018-be0b-4502-9562-b0999cdc8245\" (UID: \"330ac018-be0b-4502-9562-b0999cdc8245\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797780 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-envoy\") pod \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797818 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-kube-api-access-hcnm9\" (UniqueName: \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-kube-api-access-hcnm9\") pod \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797870 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istiod-ca-cert\") pod \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797912 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-token\" (UniqueName: \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-token\") pod \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\" (UID: \"3dbf3aa0-4a0b-4069-b219-6f73873bf995\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.797955 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-certs\") pod \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\" (UID: \"6691f77d-0336-4b20-a16f-9d8b5f2159e2\") " Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799128 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799150 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799163 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799176 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c48728-a545-4f28-a74a-99f701418e9d-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799189 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799202 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799216 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799230 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799258 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799273 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799287 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799300 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799315 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.809130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.799529 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-certs" pod "6691f77d-0336-4b20-a16f-9d8b5f2159e2" (UID: "6691f77d-0336-4b20-a16f-9d8b5f2159e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.841191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.840451 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-credential-socket" pod "6691f77d-0336-4b20-a16f-9d8b5f2159e2" (UID: "6691f77d-0336-4b20-a16f-9d8b5f2159e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.841191 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.840870 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330ac018-be0b-4502-9562-b0999cdc8245-istiod-ca-cert" pod "330ac018-be0b-4502-9562-b0999cdc8245" (UID: "330ac018-be0b-4502-9562-b0999cdc8245"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:26.849396 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.848979 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-data" pod "330ac018-be0b-4502-9562-b0999cdc8245" (UID: "330ac018-be0b-4502-9562-b0999cdc8245"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.855079 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.854573 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istiod-ca-cert" pod "6691f77d-0336-4b20-a16f-9d8b5f2159e2" (UID: "6691f77d-0336-4b20-a16f-9d8b5f2159e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:26.855079 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.855004 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-socket" pod "6691f77d-0336-4b20-a16f-9d8b5f2159e2" (UID: "6691f77d-0336-4b20-a16f-9d8b5f2159e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.889562 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.880199 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-credential-socket" pod "330ac018-be0b-4502-9562-b0999cdc8245" (UID: "330ac018-be0b-4502-9562-b0999cdc8245"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.902538 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.902172 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-certs" pod "330ac018-be0b-4502-9562-b0999cdc8245" (UID: "330ac018-be0b-4502-9562-b0999cdc8245"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.951573 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.951377 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-socket" pod "3dbf3aa0-4a0b-4069-b219-6f73873bf995" (UID: "3dbf3aa0-4a0b-4069-b219-6f73873bf995"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.959381 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.958526 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-c6dxg"] Nov 01 22:44:26.970985 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.970710 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istiod-ca-cert" pod "3dbf3aa0-4a0b-4069-b219-6f73873bf995" (UID: "3dbf3aa0-4a0b-4069-b219-6f73873bf995"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:26.971655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.971229 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-socket" pod "330ac018-be0b-4502-9562-b0999cdc8245" (UID: "330ac018-be0b-4502-9562-b0999cdc8245"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.971655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.971355 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/330ac018-be0b-4502-9562-b0999cdc8245-istio-podinfo" pod "330ac018-be0b-4502-9562-b0999cdc8245" (UID: "330ac018-be0b-4502-9562-b0999cdc8245"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:27.047662 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:26.997161 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-data" pod "6691f77d-0336-4b20-a16f-9d8b5f2159e2" (UID: "6691f77d-0336-4b20-a16f-9d8b5f2159e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:27.054157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.054099 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.131263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.131230 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.131505 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.131486 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.131608 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.131595 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.131692 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.131682 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.131778 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.131767 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330ac018-be0b-4502-9562-b0999cdc8245-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.131872 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.131861 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.131954 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.131943 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.132030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.132020 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.132141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.132129 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.132259 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.132247 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.132364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.132353 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.132465 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.132454 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330ac018-be0b-4502-9562-b0999cdc8245-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.173394 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.173326 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-envoy" pod "3dbf3aa0-4a0b-4069-b219-6f73873bf995" (UID: "3dbf3aa0-4a0b-4069-b219-6f73873bf995"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:27.234186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.233474 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-envoy" pod "6691f77d-0336-4b20-a16f-9d8b5f2159e2" (UID: "6691f77d-0336-4b20-a16f-9d8b5f2159e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:27.237921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.237473 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-token" pod "3dbf3aa0-4a0b-4069-b219-6f73873bf995" (UID: "3dbf3aa0-4a0b-4069-b219-6f73873bf995"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:27.256372 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.256339 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.260966 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.256593 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.260966 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.256619 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.264494 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.263365 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-istio-token" pod "330ac018-be0b-4502-9562-b0999cdc8245" (UID: "330ac018-be0b-4502-9562-b0999cdc8245"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:27.264494 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.263779 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-token" pod "6691f77d-0336-4b20-a16f-9d8b5f2159e2" (UID: "6691f77d-0336-4b20-a16f-9d8b5f2159e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:27.264494 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.264345 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-kube-api-access-r4h8q" pod "330ac018-be0b-4502-9562-b0999cdc8245" (UID: "330ac018-be0b-4502-9562-b0999cdc8245"). InnerVolumeSpecName "kube-api-access-r4h8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:27.264494 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.264415 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-envoy" pod "330ac018-be0b-4502-9562-b0999cdc8245" (UID: "330ac018-be0b-4502-9562-b0999cdc8245"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:27.306597 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.306544 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-kube-api-access-hcnm9" pod "3dbf3aa0-4a0b-4069-b219-6f73873bf995" (UID: "3dbf3aa0-4a0b-4069-b219-6f73873bf995"). InnerVolumeSpecName "kube-api-access-hcnm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:27.319488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.307259 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-podinfo" pod "3dbf3aa0-4a0b-4069-b219-6f73873bf995" (UID: "3dbf3aa0-4a0b-4069-b219-6f73873bf995"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:27.319488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.307430 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27c48728-a545-4f28-a74a-99f701418e9d-istio-podinfo" pod "27c48728-a545-4f28-a74a-99f701418e9d" (UID: "27c48728-a545-4f28-a74a-99f701418e9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:27.319488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.307601 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-token" pod "eb948b12-1e45-4e5f-9750-eed6b5ad09cf" (UID: "eb948b12-1e45-4e5f-9750-eed6b5ad09cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:27.329475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.329434 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-envoy" pod "27c48728-a545-4f28-a74a-99f701418e9d" (UID: "27c48728-a545-4f28-a74a-99f701418e9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:27.330078 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.329829 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-kube-api-access-6wmrm" pod "eb948b12-1e45-4e5f-9750-eed6b5ad09cf" (UID: "eb948b12-1e45-4e5f-9750-eed6b5ad09cf"). InnerVolumeSpecName "kube-api-access-6wmrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:27.330327 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.330300 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-podinfo" pod "6691f77d-0336-4b20-a16f-9d8b5f2159e2" (UID: "6691f77d-0336-4b20-a16f-9d8b5f2159e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:27.330556 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.330539 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-kube-api-access-5wmhg" pod "27c48728-a545-4f28-a74a-99f701418e9d" (UID: "27c48728-a545-4f28-a74a-99f701418e9d"). InnerVolumeSpecName "kube-api-access-5wmhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:27.330715 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.330700 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-podinfo" pod "eb948b12-1e45-4e5f-9750-eed6b5ad09cf" (UID: "eb948b12-1e45-4e5f-9750-eed6b5ad09cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:27.330812 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.330748 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-istio-token" pod "27c48728-a545-4f28-a74a-99f701418e9d" (UID: "27c48728-a545-4f28-a74a-99f701418e9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:27.330901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.330872 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-envoy" pod "eb948b12-1e45-4e5f-9750-eed6b5ad09cf" (UID: "eb948b12-1e45-4e5f-9750-eed6b5ad09cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:27.330966 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.330826 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-kube-api-access-rsfsv" pod "6691f77d-0336-4b20-a16f-9d8b5f2159e2" (UID: "6691f77d-0336-4b20-a16f-9d8b5f2159e2"). InnerVolumeSpecName "kube-api-access-rsfsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:27.332782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.332758 6466 scope.go:122] "RemoveContainer" containerID="5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb" Nov 01 22:44:27.334308 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:44:27.334277 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb\": container with ID starting with 5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb not found: ID does not exist" containerID="5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb" Nov 01 22:44:27.334391 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.334318 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb"} err="failed to get container status \"5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb\": rpc error: code = NotFound desc = could not find container \"5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb\": container with ID starting with 5710ebdf0e73d701b301f62ef3051329df6f2f571d74dc0acd9dd0462528d3eb not found: ID does not exist" Nov 01 22:44:27.334391 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.334345 6466 scope.go:122] "RemoveContainer" containerID="a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8" Nov 01 22:44:27.348300 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.348279 6466 scope.go:122] "RemoveContainer" containerID="a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8" Nov 01 22:44:27.348648 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:44:27.348628 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8\": container with ID starting with a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8 not found: ID does not exist" containerID="a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8" Nov 01 22:44:27.348714 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.348658 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8"} err="failed to get container status \"a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8\": rpc error: code = NotFound desc = could not find container \"a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8\": container with ID starting with a3746b1b22edb10fc31b5890dfe5a0e9296bdc6d71b41daad79d0ce0118848c8 not found: ID does not exist" Nov 01 22:44:27.357935 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.357906 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.357935 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.357934 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbf3aa0-4a0b-4069-b219-6f73873bf995-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.357947 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c48728-a545-4f28-a74a-99f701418e9d-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.357957 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330ac018-be0b-4502-9562-b0999cdc8245-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.357966 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.357975 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c48728-a545-4f28-a74a-99f701418e9d-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.357984 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4h8q\" (UniqueName: \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-kube-api-access-r4h8q\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.357993 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330ac018-be0b-4502-9562-b0999cdc8245-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.358002 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsfsv\" (UniqueName: \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-kube-api-access-rsfsv\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.358010 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcnm9\" (UniqueName: \"kubernetes.io/projected/3dbf3aa0-4a0b-4069-b219-6f73873bf995-kube-api-access-hcnm9\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.358019 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wmhg\" (UniqueName: \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-kube-api-access-5wmhg\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.358028 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.358036 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c48728-a545-4f28-a74a-99f701418e9d-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.358044 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wmrm\" (UniqueName: \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-kube-api-access-6wmrm\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.358053 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6691f77d-0336-4b20-a16f-9d8b5f2159e2-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.358131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.358061 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb948b12-1e45-4e5f-9750-eed6b5ad09cf-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:27.762121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.762033 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7"] Nov 01 22:44:27.885175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.788454 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" Nov 01 22:44:27.885175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.799631 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv" event={"ID":"6691f77d-0336-4b20-a16f-9d8b5f2159e2","Type":"ContainerDied","Data":"474921c12d9e7ed8540e67a58dc9ee838c368a244af7bee1aa8a6915e6842303"} Nov 01 22:44:27.885175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.799713 6466 scope.go:122] "RemoveContainer" containerID="dc53a77f9fc6526d7974b64406175f2742531b105f6805db539b2ab96c174ce9" Nov 01 22:44:27.885175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.858986 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" event={"ID":"330ac018-be0b-4502-9562-b0999cdc8245","Type":"ContainerDied","Data":"ae9205f3d874247f5904200ad93aff3190a681dde631bde486ae167b8f239aed"} Nov 01 22:44:27.885175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.859154 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x" Nov 01 22:44:27.941144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.932784 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-z2dw7"] Nov 01 22:44:27.951194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.951146 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" event={"ID":"3dbf3aa0-4a0b-4069-b219-6f73873bf995","Type":"ContainerDied","Data":"b824392195915f30f35bef044aaf67178045bc132ac3a59d39fb92f07a4fbabf"} Nov 01 22:44:27.951549 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:27.951524 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz" Nov 01 22:44:28.043953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.043926 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq"] Nov 01 22:44:28.199132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.198183 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-4g5bq"] Nov 01 22:44:28.229151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.215981 6466 scope.go:122] "RemoveContainer" containerID="4edc1e58707ec0d31e717b79207875d54725dd8d967879d007689fe353db237d" Nov 01 22:44:28.264472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.264441 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x"] Nov 01 22:44:28.267785 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.267753 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-6cq4x"] Nov 01 22:44:28.293910 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.293160 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv"] Nov 01 22:44:28.307860 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.307836 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-bw5jv"] Nov 01 22:44:28.308376 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.308359 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz"] Nov 01 22:44:28.314130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.314034 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-8xthz"] Nov 01 22:44:28.358166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.358132 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c48728-a545-4f28-a74a-99f701418e9d" path="/var/lib/kubelet/pods/27c48728-a545-4f28-a74a-99f701418e9d/volumes" Nov 01 22:44:28.359198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.359176 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330ac018-be0b-4502-9562-b0999cdc8245" path="/var/lib/kubelet/pods/330ac018-be0b-4502-9562-b0999cdc8245/volumes" Nov 01 22:44:28.360044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.360023 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" path="/var/lib/kubelet/pods/3dbf3aa0-4a0b-4069-b219-6f73873bf995/volumes" Nov 01 22:44:28.361429 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.361408 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" path="/var/lib/kubelet/pods/6691f77d-0336-4b20-a16f-9d8b5f2159e2/volumes" Nov 01 22:44:28.362320 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.362301 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" path="/var/lib/kubelet/pods/84e4b70b-8d5e-4d95-9a62-3996d9fac057/volumes" Nov 01 22:44:28.363147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.363099 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" path="/var/lib/kubelet/pods/eb948b12-1e45-4e5f-9750-eed6b5ad09cf/volumes" Nov 01 22:44:28.422100 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:28.422076 6466 scope.go:122] "RemoveContainer" containerID="ede69a7260c2703129e4749bc566c595b8fc73843a03869a4bb1dd7256224d1b" Nov 01 22:44:31.233133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:31.233085 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk"] Nov 01 22:44:31.301128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:31.300918 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" podUID="68f2b87e-7e90-47d7-b323-26bbf215051b" containerName="payload-processing" containerID="cri-o://44e758b8a7582ba22ffaf84bd02b819c7823d2bed979b5c266fc357de366d6bd" gracePeriod=25 Nov 01 22:44:31.396326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:31.396293 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:44:31.540426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:31.540342 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" podUID="e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6" containerName="payload-pre-processing" containerID="cri-o://64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff" gracePeriod=25 Nov 01 22:44:31.980687 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:31.980643 6466 generic.go:356] "Generic (PLEG): container finished" podID="68f2b87e-7e90-47d7-b323-26bbf215051b" containerID="44e758b8a7582ba22ffaf84bd02b819c7823d2bed979b5c266fc357de366d6bd" exitCode=0 Nov 01 22:44:31.980943 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:31.980920 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" event={"ID":"68f2b87e-7e90-47d7-b323-26bbf215051b","Type":"ContainerDied","Data":"44e758b8a7582ba22ffaf84bd02b819c7823d2bed979b5c266fc357de366d6bd"} Nov 01 22:44:31.981208 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:31.981187 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:44:32.071102 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.071069 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f2b87e-7e90-47d7-b323-26bbf215051b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f2b87e-7e90-47d7-b323-26bbf215051b-plugins-config-volume\") pod \"68f2b87e-7e90-47d7-b323-26bbf215051b\" (UID: \"68f2b87e-7e90-47d7-b323-26bbf215051b\") " Nov 01 22:44:32.071267 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.071221 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f2b87e-7e90-47d7-b323-26bbf215051b-kube-api-access-jwltl\" (UniqueName: \"kubernetes.io/projected/68f2b87e-7e90-47d7-b323-26bbf215051b-kube-api-access-jwltl\") pod \"68f2b87e-7e90-47d7-b323-26bbf215051b\" (UID: \"68f2b87e-7e90-47d7-b323-26bbf215051b\") " Nov 01 22:44:32.071515 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.071485 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f2b87e-7e90-47d7-b323-26bbf215051b-plugins-config-volume" pod "68f2b87e-7e90-47d7-b323-26bbf215051b" (UID: "68f2b87e-7e90-47d7-b323-26bbf215051b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:32.072264 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.072239 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f2b87e-7e90-47d7-b323-26bbf215051b-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:32.073861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.073837 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f2b87e-7e90-47d7-b323-26bbf215051b-kube-api-access-jwltl" pod "68f2b87e-7e90-47d7-b323-26bbf215051b" (UID: "68f2b87e-7e90-47d7-b323-26bbf215051b"). InnerVolumeSpecName "kube-api-access-jwltl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:32.149074 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.149046 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:44:32.173474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.173445 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-plugins-config-volume\") pod \"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6\" (UID: \"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6\") " Nov 01 22:44:32.173644 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.173630 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-kube-api-access-tqpq5\" (UniqueName: \"kubernetes.io/projected/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-kube-api-access-tqpq5\") pod \"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6\" (UID: \"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6\") " Nov 01 22:44:32.173858 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.173833 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-plugins-config-volume" pod "e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6" (UID: "e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:32.174273 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.174254 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:32.174358 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.174277 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwltl\" (UniqueName: \"kubernetes.io/projected/68f2b87e-7e90-47d7-b323-26bbf215051b-kube-api-access-jwltl\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:32.175643 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.175624 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-kube-api-access-tqpq5" pod "e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6" (UID: "e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6"). InnerVolumeSpecName "kube-api-access-tqpq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:32.286309 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.286221 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqpq5\" (UniqueName: \"kubernetes.io/projected/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6-kube-api-access-tqpq5\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:32.991289 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.991251 6466 generic.go:356] "Generic (PLEG): container finished" podID="e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6" containerID="64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff" exitCode=0 Nov 01 22:44:32.991464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.991306 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" Nov 01 22:44:32.991464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.991331 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" event={"ID":"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6","Type":"ContainerDied","Data":"64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff"} Nov 01 22:44:32.991464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.991418 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8" event={"ID":"e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6","Type":"ContainerDied","Data":"916f9c9061640db8984f6ad00c7c7835b39e466dc05112cfe628ad975a5c16c5"} Nov 01 22:44:32.991464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.991449 6466 scope.go:122] "RemoveContainer" containerID="64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff" Nov 01 22:44:32.993245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.993221 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" event={"ID":"68f2b87e-7e90-47d7-b323-26bbf215051b","Type":"ContainerDied","Data":"689dc429a6ef07b7fb45f58c2e631ef75717fe86f0366661449285c69e059bfe"} Nov 01 22:44:32.993380 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:32.993235 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5" Nov 01 22:44:33.041135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:33.038135 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8"] Nov 01 22:44:33.057133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:33.048448 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b4f6bb-5b4fbdff-mfjv8"] Nov 01 22:44:33.057133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:33.056209 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5"] Nov 01 22:44:33.076389 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:33.065939 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b4f6bb-6f8c8d55d9-ltvm5"] Nov 01 22:44:33.076389 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:33.067368 6466 scope.go:122] "RemoveContainer" containerID="64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff" Nov 01 22:44:33.076389 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:44:33.068248 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff\": container with ID starting with 64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff not found: ID does not exist" containerID="64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff" Nov 01 22:44:33.076389 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:33.068292 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff"} err="failed to get container status \"64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff\": rpc error: code = NotFound desc = could not find container \"64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff\": container with ID starting with 64f95cf542da3c18bcad1fe97b4e09640f9f24c7f39b088827adef01fc0fb5ff not found: ID does not exist" Nov 01 22:44:33.076389 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:33.068318 6466 scope.go:122] "RemoveContainer" containerID="44e758b8a7582ba22ffaf84bd02b819c7823d2bed979b5c266fc357de366d6bd" Nov 01 22:44:34.343820 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:34.343785 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f2b87e-7e90-47d7-b323-26bbf215051b" path="/var/lib/kubelet/pods/68f2b87e-7e90-47d7-b323-26bbf215051b/volumes" Nov 01 22:44:34.344534 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:34.344511 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6" path="/var/lib/kubelet/pods/e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6/volumes" Nov 01 22:44:36.234172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.234134 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" podUID="70c6daab-48e9-4de4-b639-90f212867977" containerName="payload-pre-processing" containerID="cri-o://b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504" gracePeriod=25 Nov 01 22:44:36.395567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.395532 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq"] Nov 01 22:44:36.396797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396767 6466 cpu_manager.go:414] "containerMap: removing container" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" Nov 01 22:44:36.396797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396793 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" Nov 01 22:44:36.397004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396807 6466 cpu_manager.go:414] "containerMap: removing container" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" Nov 01 22:44:36.397004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396812 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" Nov 01 22:44:36.397004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396905 6466 cpu_manager.go:414] "containerMap: removing container" podUID="e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6" containerName="payload-pre-processing" Nov 01 22:44:36.397004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396914 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6" containerName="payload-pre-processing" Nov 01 22:44:36.397004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396931 6466 cpu_manager.go:414] "containerMap: removing container" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" Nov 01 22:44:36.397004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396939 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" Nov 01 22:44:36.397004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396951 6466 cpu_manager.go:414] "containerMap: removing container" podUID="68f2b87e-7e90-47d7-b323-26bbf215051b" containerName="payload-processing" Nov 01 22:44:36.397004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396958 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f2b87e-7e90-47d7-b323-26bbf215051b" containerName="payload-processing" Nov 01 22:44:36.397004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396980 6466 cpu_manager.go:414] "containerMap: removing container" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" Nov 01 22:44:36.397004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.396987 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397016 6466 cpu_manager.go:414] "containerMap: removing container" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397025 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397045 6466 cpu_manager.go:414] "containerMap: removing container" podUID="27c48728-a545-4f28-a74a-99f701418e9d" containerName="istio-proxy" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397052 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c48728-a545-4f28-a74a-99f701418e9d" containerName="istio-proxy" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397302 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f4c1f0-15be-4c76-9bd2-ea3b359b25d6" containerName="payload-pre-processing" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397320 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="6691f77d-0336-4b20-a16f-9d8b5f2159e2" containerName="istio-proxy" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397331 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f2b87e-7e90-47d7-b323-26bbf215051b" containerName="payload-processing" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397347 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="330ac018-be0b-4502-9562-b0999cdc8245" containerName="istio-proxy" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397360 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbf3aa0-4a0b-4069-b219-6f73873bf995" containerName="istio-proxy" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397370 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c48728-a545-4f28-a74a-99f701418e9d" containerName="istio-proxy" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397385 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e4b70b-8d5e-4d95-9a62-3996d9fac057" containerName="istio-proxy" Nov 01 22:44:36.397520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.397394 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb948b12-1e45-4e5f-9750-eed6b5ad09cf" containerName="istio-proxy" Nov 01 22:44:36.402633 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.402582 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.410704 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.410653 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ae0e3a-serving-cert\"" Nov 01 22:44:36.417305 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.417271 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq"] Nov 01 22:44:36.532512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.528549 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-maas-api-tls\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.532512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.528633 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhcn4\" (UniqueName: \"kubernetes.io/projected/402f118e-e4ef-4bfe-9e0f-ed5003167964-kube-api-access-hhcn4\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.532512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.528692 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-metrics-certs\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.532512 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.528982 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f118e-e4ef-4bfe-9e0f-ed5003167964-openshift-service-ca\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.659140 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.656444 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f118e-e4ef-4bfe-9e0f-ed5003167964-openshift-service-ca\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.682377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.655305 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f118e-e4ef-4bfe-9e0f-ed5003167964-openshift-service-ca\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.682377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.675525 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-maas-api-tls\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.682377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.675605 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhcn4\" (UniqueName: \"kubernetes.io/projected/402f118e-e4ef-4bfe-9e0f-ed5003167964-kube-api-access-hhcn4\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.682377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.675655 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-metrics-certs\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.725132 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:44:36.685187 6466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-serving-cert: secret "maas-api-e2e-api-b-ae0e3a-serving-cert" not found Nov 01 22:44:36.725132 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:44:36.685273 6466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-maas-api-tls podName:402f118e-e4ef-4bfe-9e0f-ed5003167964 nodeName:}" failed. No retries permitted until 2026-11-01 22:44:37.185250291 +0000 UTC m=+1280.969689214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-maas-api-tls") pod "maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" (UID: "402f118e-e4ef-4bfe-9e0f-ed5003167964") : secret "maas-api-e2e-api-b-ae0e3a-serving-cert" not found Nov 01 22:44:36.735941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.735901 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhcn4\" (UniqueName: \"kubernetes.io/projected/402f118e-e4ef-4bfe-9e0f-ed5003167964-kube-api-access-hhcn4\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.736623 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.736594 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-metrics-certs\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:36.742678 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.742304 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:44:36.777728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.777683 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c6daab-48e9-4de4-b639-90f212867977-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c6daab-48e9-4de4-b639-90f212867977-plugins-config-volume\") pod \"70c6daab-48e9-4de4-b639-90f212867977\" (UID: \"70c6daab-48e9-4de4-b639-90f212867977\") " Nov 01 22:44:36.777899 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.777775 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c6daab-48e9-4de4-b639-90f212867977-kube-api-access-ljnj4\" (UniqueName: \"kubernetes.io/projected/70c6daab-48e9-4de4-b639-90f212867977-kube-api-access-ljnj4\") pod \"70c6daab-48e9-4de4-b639-90f212867977\" (UID: \"70c6daab-48e9-4de4-b639-90f212867977\") " Nov 01 22:44:36.778237 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.778208 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c6daab-48e9-4de4-b639-90f212867977-plugins-config-volume" pod "70c6daab-48e9-4de4-b639-90f212867977" (UID: "70c6daab-48e9-4de4-b639-90f212867977"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:36.778897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.778868 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c6daab-48e9-4de4-b639-90f212867977-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:36.780063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.780041 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c6daab-48e9-4de4-b639-90f212867977-kube-api-access-ljnj4" pod "70c6daab-48e9-4de4-b639-90f212867977" (UID: "70c6daab-48e9-4de4-b639-90f212867977"). InnerVolumeSpecName "kube-api-access-ljnj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:36.880673 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:36.880595 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljnj4\" (UniqueName: \"kubernetes.io/projected/70c6daab-48e9-4de4-b639-90f212867977-kube-api-access-ljnj4\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:37.080441 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.080401 6466 generic.go:356] "Generic (PLEG): container finished" podID="70c6daab-48e9-4de4-b639-90f212867977" containerID="b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504" exitCode=0 Nov 01 22:44:37.080751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.080527 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" event={"ID":"70c6daab-48e9-4de4-b639-90f212867977","Type":"ContainerDied","Data":"b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504"} Nov 01 22:44:37.080923 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.080906 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" event={"ID":"70c6daab-48e9-4de4-b639-90f212867977","Type":"ContainerDied","Data":"5319c8471ea6d279565a254a350151cb40156ff9597b50b120a8e2cfb30d6290"} Nov 01 22:44:37.081037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.081025 6466 scope.go:122] "RemoveContainer" containerID="b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504" Nov 01 22:44:37.081252 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.080627 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk" Nov 01 22:44:37.172297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.172261 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk"] Nov 01 22:44:37.187166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.183796 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b7320-5f768959bc-qc5kk"] Nov 01 22:44:37.203450 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.203411 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-maas-api-tls\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:37.228137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.226757 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-maas-api-tls\") pod \"maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:37.235048 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.232526 6466 scope.go:122] "RemoveContainer" containerID="b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504" Nov 01 22:44:37.238280 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:44:37.238023 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504\": container with ID starting with b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504 not found: ID does not exist" containerID="b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504" Nov 01 22:44:37.238280 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.238060 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504"} err="failed to get container status \"b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504\": rpc error: code = NotFound desc = could not find container \"b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504\": container with ID starting with b87540a99862a0338900e9c77579272243b262466396c2c9503b92dc474e9504 not found: ID does not exist" Nov 01 22:44:37.332857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.332825 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:37.626476 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:37.626449 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq"] Nov 01 22:44:37.627160 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:44:37.627132 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402f118e_e4ef_4bfe_9e0f_ed5003167964.slice/crio-b246e7a2313df663c082cd5a57f6ff947d2058804c242a5f70b89755633e1e1e WatchSource:0}: Error finding container b246e7a2313df663c082cd5a57f6ff947d2058804c242a5f70b89755633e1e1e: Status 404 returned error can't find the container with id b246e7a2313df663c082cd5a57f6ff947d2058804c242a5f70b89755633e1e1e Nov 01 22:44:38.090121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:38.090072 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" event={"ID":"402f118e-e4ef-4bfe-9e0f-ed5003167964","Type":"ContainerStarted","Data":"b246e7a2313df663c082cd5a57f6ff947d2058804c242a5f70b89755633e1e1e"} Nov 01 22:44:38.343624 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:38.343546 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c6daab-48e9-4de4-b639-90f212867977" path="/var/lib/kubelet/pods/70c6daab-48e9-4de4-b639-90f212867977/volumes" Nov 01 22:44:39.100652 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:39.100617 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" event={"ID":"402f118e-e4ef-4bfe-9e0f-ed5003167964","Type":"ContainerStarted","Data":"05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21"} Nov 01 22:44:39.100850 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:39.100707 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:39.120716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:39.120657 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" podStartSLOduration=2.709659672 podStartE2EDuration="3.12064209s" totalImagesPullingTime="410.982418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:39.118912276 +0000 UTC m=+1282.903351220" watchObservedRunningTime="2026-11-01 22:44:39.12064209 +0000 UTC m=+1282.905081011" Nov 01 22:44:45.115370 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:45.115334 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:44:47.989133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:47.989011 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg"] Nov 01 22:44:47.989620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:47.989324 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" podUID="2bd09053-98bc-456a-9b80-9b32f584d2b1" containerName="maas-api" containerID="cri-o://23b19573cfd3bf67acc22d752d5e4f0a6f7c0db8194c235e824bbeb5ae24c382" gracePeriod=30 Nov 01 22:44:48.649141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.648987 6466 generic.go:356] "Generic (PLEG): container finished" podID="2bd09053-98bc-456a-9b80-9b32f584d2b1" containerID="23b19573cfd3bf67acc22d752d5e4f0a6f7c0db8194c235e824bbeb5ae24c382" exitCode=0 Nov 01 22:44:48.662772 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.649323 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" event={"ID":"2bd09053-98bc-456a-9b80-9b32f584d2b1","Type":"ContainerDied","Data":"23b19573cfd3bf67acc22d752d5e4f0a6f7c0db8194c235e824bbeb5ae24c382"} Nov 01 22:44:48.684220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.684192 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:44:48.764524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.764485 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-metrics-certs\") pod \"2bd09053-98bc-456a-9b80-9b32f584d2b1\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " Nov 01 22:44:48.764712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.764638 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-maas-api-tls\") pod \"2bd09053-98bc-456a-9b80-9b32f584d2b1\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " Nov 01 22:44:48.764761 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.764710 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd09053-98bc-456a-9b80-9b32f584d2b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bd09053-98bc-456a-9b80-9b32f584d2b1-openshift-service-ca\") pod \"2bd09053-98bc-456a-9b80-9b32f584d2b1\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " Nov 01 22:44:48.764761 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.764738 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd09053-98bc-456a-9b80-9b32f584d2b1-kube-api-access-whlwj\" (UniqueName: \"kubernetes.io/projected/2bd09053-98bc-456a-9b80-9b32f584d2b1-kube-api-access-whlwj\") pod \"2bd09053-98bc-456a-9b80-9b32f584d2b1\" (UID: \"2bd09053-98bc-456a-9b80-9b32f584d2b1\") " Nov 01 22:44:48.765127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.765083 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd09053-98bc-456a-9b80-9b32f584d2b1-openshift-service-ca" pod "2bd09053-98bc-456a-9b80-9b32f584d2b1" (UID: "2bd09053-98bc-456a-9b80-9b32f584d2b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:48.765460 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.765436 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bd09053-98bc-456a-9b80-9b32f584d2b1-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:48.766649 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.766623 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-metrics-certs" pod "2bd09053-98bc-456a-9b80-9b32f584d2b1" (UID: "2bd09053-98bc-456a-9b80-9b32f584d2b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:48.766839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.766816 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-maas-api-tls" pod "2bd09053-98bc-456a-9b80-9b32f584d2b1" (UID: "2bd09053-98bc-456a-9b80-9b32f584d2b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:48.766839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.766828 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd09053-98bc-456a-9b80-9b32f584d2b1-kube-api-access-whlwj" pod "2bd09053-98bc-456a-9b80-9b32f584d2b1" (UID: "2bd09053-98bc-456a-9b80-9b32f584d2b1"). InnerVolumeSpecName "kube-api-access-whlwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:48.867320 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.867287 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:48.867320 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.867318 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whlwj\" (UniqueName: \"kubernetes.io/projected/2bd09053-98bc-456a-9b80-9b32f584d2b1-kube-api-access-whlwj\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:48.867516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:48.867331 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd09053-98bc-456a-9b80-9b32f584d2b1-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:49.282851 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:44:49.282813 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:44:49.597526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:49.597447 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp"] Nov 01 22:44:49.613076 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:49.613046 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47"] Nov 01 22:44:49.663464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:49.662698 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" event={"ID":"2bd09053-98bc-456a-9b80-9b32f584d2b1","Type":"ContainerDied","Data":"9e2bd4fa8d8765666dd59640872c0857a40035dc706248617cc233f91e72754e"} Nov 01 22:44:49.663464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:49.662786 6466 scope.go:122] "RemoveContainer" containerID="23b19573cfd3bf67acc22d752d5e4f0a6f7c0db8194c235e824bbeb5ae24c382" Nov 01 22:44:49.663464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:49.662719 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg" Nov 01 22:44:49.691570 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:49.691543 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg"] Nov 01 22:44:49.697129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:49.697092 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9b7320-598fdbd449-cg7sg"] Nov 01 22:44:50.343176 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:50.343146 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd09053-98bc-456a-9b80-9b32f584d2b1" path="/var/lib/kubelet/pods/2bd09053-98bc-456a-9b80-9b32f584d2b1/volumes" Nov 01 22:44:54.598446 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:54.598393 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" podUID="516adbc4-8ec9-42a2-8c3b-868e5bd73c9a" containerName="payload-processing" containerID="cri-o://6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73" gracePeriod=25 Nov 01 22:44:54.614129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:54.614067 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" podUID="de632f88-68c6-4e45-bef6-301d8350dd60" containerName="payload-pre-processing" containerID="cri-o://21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915" gracePeriod=25 Nov 01 22:44:55.330044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.329660 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:44:55.424030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.423994 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:44:55.450001 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.449974 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-plugins-config-volume\") pod \"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a\" (UID: \"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a\") " Nov 01 22:44:55.450192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.450177 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-kube-api-access-8qd6l\" (UniqueName: \"kubernetes.io/projected/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-kube-api-access-8qd6l\") pod \"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a\" (UID: \"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a\") " Nov 01 22:44:55.450398 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.450376 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-plugins-config-volume" pod "516adbc4-8ec9-42a2-8c3b-868e5bd73c9a" (UID: "516adbc4-8ec9-42a2-8c3b-868e5bd73c9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:55.450895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.450876 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:55.452414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.452390 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-kube-api-access-8qd6l" pod "516adbc4-8ec9-42a2-8c3b-868e5bd73c9a" (UID: "516adbc4-8ec9-42a2-8c3b-868e5bd73c9a"). InnerVolumeSpecName "kube-api-access-8qd6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:55.552000 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.551918 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de632f88-68c6-4e45-bef6-301d8350dd60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de632f88-68c6-4e45-bef6-301d8350dd60-plugins-config-volume\") pod \"de632f88-68c6-4e45-bef6-301d8350dd60\" (UID: \"de632f88-68c6-4e45-bef6-301d8350dd60\") " Nov 01 22:44:55.552301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.552283 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de632f88-68c6-4e45-bef6-301d8350dd60-kube-api-access-6c4pd\" (UniqueName: \"kubernetes.io/projected/de632f88-68c6-4e45-bef6-301d8350dd60-kube-api-access-6c4pd\") pod \"de632f88-68c6-4e45-bef6-301d8350dd60\" (UID: \"de632f88-68c6-4e45-bef6-301d8350dd60\") " Nov 01 22:44:55.552369 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.552321 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de632f88-68c6-4e45-bef6-301d8350dd60-plugins-config-volume" pod "de632f88-68c6-4e45-bef6-301d8350dd60" (UID: "de632f88-68c6-4e45-bef6-301d8350dd60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:55.552850 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.552834 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de632f88-68c6-4e45-bef6-301d8350dd60-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:55.552889 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.552858 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qd6l\" (UniqueName: \"kubernetes.io/projected/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a-kube-api-access-8qd6l\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:55.554428 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.554406 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de632f88-68c6-4e45-bef6-301d8350dd60-kube-api-access-6c4pd" pod "de632f88-68c6-4e45-bef6-301d8350dd60" (UID: "de632f88-68c6-4e45-bef6-301d8350dd60"). InnerVolumeSpecName "kube-api-access-6c4pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:55.654121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.654084 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c4pd\" (UniqueName: \"kubernetes.io/projected/de632f88-68c6-4e45-bef6-301d8350dd60-kube-api-access-6c4pd\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:55.716550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.716517 6466 generic.go:356] "Generic (PLEG): container finished" podID="516adbc4-8ec9-42a2-8c3b-868e5bd73c9a" containerID="6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73" exitCode=0 Nov 01 22:44:55.716746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.716571 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" Nov 01 22:44:55.716746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.716593 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" event={"ID":"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a","Type":"ContainerDied","Data":"6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73"} Nov 01 22:44:55.716746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.716647 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp" event={"ID":"516adbc4-8ec9-42a2-8c3b-868e5bd73c9a","Type":"ContainerDied","Data":"faf66a17671904f4b796de6ff5e3483b3234b66a82a495b0bf14441aab493143"} Nov 01 22:44:55.716746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.716668 6466 scope.go:122] "RemoveContainer" containerID="6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73" Nov 01 22:44:55.718257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.718237 6466 generic.go:356] "Generic (PLEG): container finished" podID="de632f88-68c6-4e45-bef6-301d8350dd60" containerID="21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915" exitCode=0 Nov 01 22:44:55.718360 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.718336 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" Nov 01 22:44:55.718491 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.718356 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" event={"ID":"de632f88-68c6-4e45-bef6-301d8350dd60","Type":"ContainerDied","Data":"21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915"} Nov 01 22:44:55.718491 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.718395 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47" event={"ID":"de632f88-68c6-4e45-bef6-301d8350dd60","Type":"ContainerDied","Data":"fc8d1c12fd059bf8492da4ae4a595d883427ac3b6951bcfd3cba234e15e7b611"} Nov 01 22:44:55.728947 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.728929 6466 scope.go:122] "RemoveContainer" containerID="6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73" Nov 01 22:44:55.729282 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:44:55.729260 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73\": container with ID starting with 6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73 not found: ID does not exist" containerID="6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73" Nov 01 22:44:55.729347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.729291 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73"} err="failed to get container status \"6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73\": rpc error: code = NotFound desc = could not find container \"6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73\": container with ID starting with 6524b98bcf7c11f5fbdd6fd195147fe8fc84928b6b1695a69ad0423f29dcea73 not found: ID does not exist" Nov 01 22:44:55.729347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.729308 6466 scope.go:122] "RemoveContainer" containerID="21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915" Nov 01 22:44:55.739039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.739023 6466 scope.go:122] "RemoveContainer" containerID="21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915" Nov 01 22:44:55.739298 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:44:55.739281 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915\": container with ID starting with 21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915 not found: ID does not exist" containerID="21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915" Nov 01 22:44:55.739362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.739306 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915"} err="failed to get container status \"21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915\": rpc error: code = NotFound desc = could not find container \"21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915\": container with ID starting with 21d0aa9c9e1cbb47681be44414dd4cb87c58de40384c0216020c9903bc63f915 not found: ID does not exist" Nov 01 22:44:55.745422 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.745402 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47"] Nov 01 22:44:55.749230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.749210 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3e228e-697886864c-8qr47"] Nov 01 22:44:55.784180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.784147 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp"] Nov 01 22:44:55.795565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:55.795535 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3e228e-98b4865b9-88klp"] Nov 01 22:44:56.393531 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:56.393487 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="516adbc4-8ec9-42a2-8c3b-868e5bd73c9a" path="/var/lib/kubelet/pods/516adbc4-8ec9-42a2-8c3b-868e5bd73c9a/volumes" Nov 01 22:44:56.395377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:56.394217 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de632f88-68c6-4e45-bef6-301d8350dd60" path="/var/lib/kubelet/pods/de632f88-68c6-4e45-bef6-301d8350dd60/volumes" Nov 01 22:44:57.145185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:57.145152 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k"] Nov 01 22:44:57.146230 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:57.145395 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" podUID="0bcccba9-871e-409f-80ce-269ca37ad8a0" containerName="maas-api" containerID="cri-o://bd56549a3e312887f0a83072ba63d4ddcdeff4859a6dbb14385a368c952c00a8" gracePeriod=30 Nov 01 22:44:57.932936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:57.927868 6466 generic.go:356] "Generic (PLEG): container finished" podID="0bcccba9-871e-409f-80ce-269ca37ad8a0" containerID="bd56549a3e312887f0a83072ba63d4ddcdeff4859a6dbb14385a368c952c00a8" exitCode=0 Nov 01 22:44:57.932936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:57.928182 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" event={"ID":"0bcccba9-871e-409f-80ce-269ca37ad8a0","Type":"ContainerDied","Data":"bd56549a3e312887f0a83072ba63d4ddcdeff4859a6dbb14385a368c952c00a8"} Nov 01 22:44:57.932936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:57.928434 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5"] Nov 01 22:44:57.932936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:57.928736 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" containerID="cri-o://b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78" gracePeriod=30 Nov 01 22:44:57.951035 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:57.951002 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:44:58.088192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.088154 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-maas-api-tls\") pod \"0bcccba9-871e-409f-80ce-269ca37ad8a0\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " Nov 01 22:44:58.088397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.088232 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcccba9-871e-409f-80ce-269ca37ad8a0-kube-api-access-z87dt\" (UniqueName: \"kubernetes.io/projected/0bcccba9-871e-409f-80ce-269ca37ad8a0-kube-api-access-z87dt\") pod \"0bcccba9-871e-409f-80ce-269ca37ad8a0\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " Nov 01 22:44:58.088464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.088442 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcccba9-871e-409f-80ce-269ca37ad8a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bcccba9-871e-409f-80ce-269ca37ad8a0-openshift-service-ca\") pod \"0bcccba9-871e-409f-80ce-269ca37ad8a0\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " Nov 01 22:44:58.101133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.088550 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-metrics-certs\") pod \"0bcccba9-871e-409f-80ce-269ca37ad8a0\" (UID: \"0bcccba9-871e-409f-80ce-269ca37ad8a0\") " Nov 01 22:44:58.101408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.101370 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcccba9-871e-409f-80ce-269ca37ad8a0-openshift-service-ca" pod "0bcccba9-871e-409f-80ce-269ca37ad8a0" (UID: "0bcccba9-871e-409f-80ce-269ca37ad8a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:58.170137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.159399 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-metrics-certs" pod "0bcccba9-871e-409f-80ce-269ca37ad8a0" (UID: "0bcccba9-871e-409f-80ce-269ca37ad8a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:58.170137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.163223 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-maas-api-tls" pod "0bcccba9-871e-409f-80ce-269ca37ad8a0" (UID: "0bcccba9-871e-409f-80ce-269ca37ad8a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:58.242268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.188725 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcccba9-871e-409f-80ce-269ca37ad8a0-kube-api-access-z87dt" pod "0bcccba9-871e-409f-80ce-269ca37ad8a0" (UID: "0bcccba9-871e-409f-80ce-269ca37ad8a0"). InnerVolumeSpecName "kube-api-access-z87dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:58.242268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.202892 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bcccba9-871e-409f-80ce-269ca37ad8a0-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:58.242268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.202926 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:58.242268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.202941 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bcccba9-871e-409f-80ce-269ca37ad8a0-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:58.242268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.202955 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z87dt\" (UniqueName: \"kubernetes.io/projected/0bcccba9-871e-409f-80ce-269ca37ad8a0-kube-api-access-z87dt\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:58.938511 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.938481 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" Nov 01 22:44:58.938685 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.938519 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k" event={"ID":"0bcccba9-871e-409f-80ce-269ca37ad8a0","Type":"ContainerDied","Data":"ba40882a4342acaf4f23336019b28648bef83ee946338bae79c1efed7bb29c39"} Nov 01 22:44:58.938685 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:58.938589 6466 scope.go:122] "RemoveContainer" containerID="bd56549a3e312887f0a83072ba63d4ddcdeff4859a6dbb14385a368c952c00a8" Nov 01 22:44:59.009669 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.009637 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k"] Nov 01 22:44:59.013797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.013767 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e25182-5f9775888b-t242k"] Nov 01 22:44:59.155582 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.155550 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc"] Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157095 6466 cpu_manager.go:414] "containerMap: removing container" podUID="70c6daab-48e9-4de4-b639-90f212867977" containerName="payload-pre-processing" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157154 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c6daab-48e9-4de4-b639-90f212867977" containerName="payload-pre-processing" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157169 6466 cpu_manager.go:414] "containerMap: removing container" podUID="2bd09053-98bc-456a-9b80-9b32f584d2b1" containerName="maas-api" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157177 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd09053-98bc-456a-9b80-9b32f584d2b1" containerName="maas-api" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157220 6466 cpu_manager.go:414] "containerMap: removing container" podUID="de632f88-68c6-4e45-bef6-301d8350dd60" containerName="payload-pre-processing" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157230 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de632f88-68c6-4e45-bef6-301d8350dd60" containerName="payload-pre-processing" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157246 6466 cpu_manager.go:414] "containerMap: removing container" podUID="516adbc4-8ec9-42a2-8c3b-868e5bd73c9a" containerName="payload-processing" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157251 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="516adbc4-8ec9-42a2-8c3b-868e5bd73c9a" containerName="payload-processing" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157258 6466 cpu_manager.go:414] "containerMap: removing container" podUID="0bcccba9-871e-409f-80ce-269ca37ad8a0" containerName="maas-api" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157263 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcccba9-871e-409f-80ce-269ca37ad8a0" containerName="maas-api" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157455 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c6daab-48e9-4de4-b639-90f212867977" containerName="payload-pre-processing" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157463 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd09053-98bc-456a-9b80-9b32f584d2b1" containerName="maas-api" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157474 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="de632f88-68c6-4e45-bef6-301d8350dd60" containerName="payload-pre-processing" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157482 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="516adbc4-8ec9-42a2-8c3b-868e5bd73c9a" containerName="payload-processing" Nov 01 22:44:59.158156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.157489 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcccba9-871e-409f-80ce-269ca37ad8a0" containerName="maas-api" Nov 01 22:44:59.162845 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.162825 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.171623 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.171577 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1fddbd-openshift-default-dockercfg-5zxzj\"" Nov 01 22:44:59.187613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.187587 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc"] Nov 01 22:44:59.349203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.344192 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istiod-ca-cert\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.349203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.344284 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-credential-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.349203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.344366 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-certs\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.349203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.344433 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzdw\" (UniqueName: \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-kube-api-access-5jzdw\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.349203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.344503 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-podinfo\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.349203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.344586 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-data\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.349203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.344717 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.349203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.344829 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-envoy\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.349203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.344926 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-token\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.447946 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istiod-ca-cert\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.448014 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-credential-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.448076 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-certs\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.448148 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzdw\" (UniqueName: \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-kube-api-access-5jzdw\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.448188 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-podinfo\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.448252 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-data\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.448325 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.448396 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-envoy\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.448488 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-token\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.449895 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istiod-ca-cert\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.450377 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-certs\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.453151 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.450900 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.469134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.464299 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-data\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.480138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.477507 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-credential-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.524132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.505978 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-podinfo\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.524132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.507498 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzdw\" (UniqueName: \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-kube-api-access-5jzdw\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.524132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.515727 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-token\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.548130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.539458 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-envoy\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:44:59.778642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:44:59.778607 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:45:00.070356 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.070317 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc"] Nov 01 22:45:00.070545 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:45:00.070519 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948c76bd_1965_496d_9c0d_cbb6bc453ffc.slice/crio-55f33cc262977ed6ab82fe9ab408da9b3e1ec1ef9cd94fe1f10aa6753bed1e8c WatchSource:0}: Error finding container 55f33cc262977ed6ab82fe9ab408da9b3e1ec1ef9cd94fe1f10aa6753bed1e8c: Status 404 returned error can't find the container with id 55f33cc262977ed6ab82fe9ab408da9b3e1ec1ef9cd94fe1f10aa6753bed1e8c Nov 01 22:45:00.073225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.073193 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:45:00.073302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.073258 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:45:00.073302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.073288 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:45:00.162100 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.161886 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr"] Nov 01 22:45:00.167186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.167162 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" Nov 01 22:45:00.192487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.186625 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr"] Nov 01 22:45:00.197790 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.194400 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:45:00.268081 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.268047 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp"] Nov 01 22:45:00.272919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.272897 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56"] Nov 01 22:45:00.273236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.273219 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" Nov 01 22:45:00.274056 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.274030 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssg2\" (UniqueName: \"kubernetes.io/projected/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-kube-api-access-fssg2\") pod \"maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr\" (UID: \"816a7bec-ff31-49a0-9b0c-0112e73b1a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" Nov 01 22:45:00.274207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.274081 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr\" (UID: \"816a7bec-ff31-49a0-9b0c-0112e73b1a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" Nov 01 22:45:00.287923 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.287574 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" Nov 01 22:45:00.288098 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.287934 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j"] Nov 01 22:45:00.295975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.295946 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" Nov 01 22:45:00.309156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.309102 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp"] Nov 01 22:45:00.357679 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.357642 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcccba9-871e-409f-80ce-269ca37ad8a0" path="/var/lib/kubelet/pods/0bcccba9-871e-409f-80ce-269ca37ad8a0/volumes" Nov 01 22:45:00.362829 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.362796 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf"] Nov 01 22:45:00.369440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.369413 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j"] Nov 01 22:45:00.369592 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.369573 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" Nov 01 22:45:00.374195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.373950 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56"] Nov 01 22:45:00.376809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.376767 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp\" (UID: \"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" Nov 01 22:45:00.377054 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.376848 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwb6b\" (UniqueName: \"kubernetes.io/projected/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-kube-api-access-nwb6b\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp\" (UID: \"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" Nov 01 22:45:00.377207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.377139 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssg2\" (UniqueName: \"kubernetes.io/projected/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-kube-api-access-fssg2\") pod \"maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr\" (UID: \"816a7bec-ff31-49a0-9b0c-0112e73b1a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" Nov 01 22:45:00.377207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.377181 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr\" (UID: \"816a7bec-ff31-49a0-9b0c-0112e73b1a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" Nov 01 22:45:00.378135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.378082 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr\" (UID: \"816a7bec-ff31-49a0-9b0c-0112e73b1a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" Nov 01 22:45:00.390961 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.390932 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn"] Nov 01 22:45:00.398207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.398182 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf"] Nov 01 22:45:00.408165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.398477 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" Nov 01 22:45:00.433207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.421652 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssg2\" (UniqueName: \"kubernetes.io/projected/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-kube-api-access-fssg2\") pod \"maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr\" (UID: \"816a7bec-ff31-49a0-9b0c-0112e73b1a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" Nov 01 22:45:00.448174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.447472 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf"] Nov 01 22:45:00.454823 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.454735 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" Nov 01 22:45:00.458301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.458233 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn"] Nov 01 22:45:00.470717 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.470651 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf"] Nov 01 22:45:00.480488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.480407 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9d81f5-1e85-41b2-9eac-c43ccfb98773-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf\" (UID: \"db9d81f5-1e85-41b2-9eac-c43ccfb98773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" Nov 01 22:45:00.480740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.480719 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp\" (UID: \"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" Nov 01 22:45:00.481499 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.481465 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwb6b\" (UniqueName: \"kubernetes.io/projected/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-kube-api-access-nwb6b\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp\" (UID: \"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" Nov 01 22:45:00.486185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.481631 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnfs2\" (UniqueName: \"kubernetes.io/projected/14785ed4-8e76-4509-9712-0026d2e80b74-kube-api-access-pnfs2\") pod \"maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j\" (UID: \"14785ed4-8e76-4509-9712-0026d2e80b74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" Nov 01 22:45:00.486185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.481839 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14785ed4-8e76-4509-9712-0026d2e80b74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j\" (UID: \"14785ed4-8e76-4509-9712-0026d2e80b74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" Nov 01 22:45:00.486185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.481867 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqj6\" (UniqueName: \"kubernetes.io/projected/db9d81f5-1e85-41b2-9eac-c43ccfb98773-kube-api-access-vtqj6\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf\" (UID: \"db9d81f5-1e85-41b2-9eac-c43ccfb98773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" Nov 01 22:45:00.486185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.481907 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9vgt\" (UniqueName: \"kubernetes.io/projected/5d6d8cdd-7079-4eb3-8701-d26549713a55-kube-api-access-v9vgt\") pod \"maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56\" (UID: \"5d6d8cdd-7079-4eb3-8701-d26549713a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" Nov 01 22:45:00.486185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.482042 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6d8cdd-7079-4eb3-8701-d26549713a55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56\" (UID: \"5d6d8cdd-7079-4eb3-8701-d26549713a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" Nov 01 22:45:00.486185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.483290 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp\" (UID: \"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" Nov 01 22:45:00.486185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.486059 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" Nov 01 22:45:00.502179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.501540 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwb6b\" (UniqueName: \"kubernetes.io/projected/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-kube-api-access-nwb6b\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp\" (UID: \"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" Nov 01 22:45:00.584797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.584713 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf\" (UID: \"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" Nov 01 22:45:00.584993 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.584820 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14785ed4-8e76-4509-9712-0026d2e80b74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j\" (UID: \"14785ed4-8e76-4509-9712-0026d2e80b74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" Nov 01 22:45:00.584993 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.584849 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqj6\" (UniqueName: \"kubernetes.io/projected/db9d81f5-1e85-41b2-9eac-c43ccfb98773-kube-api-access-vtqj6\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf\" (UID: \"db9d81f5-1e85-41b2-9eac-c43ccfb98773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" Nov 01 22:45:00.584993 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.584881 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vgt\" (UniqueName: \"kubernetes.io/projected/5d6d8cdd-7079-4eb3-8701-d26549713a55-kube-api-access-v9vgt\") pod \"maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56\" (UID: \"5d6d8cdd-7079-4eb3-8701-d26549713a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" Nov 01 22:45:00.584993 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.584904 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqkh\" (UniqueName: \"kubernetes.io/projected/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-kube-api-access-qqqkh\") pod \"maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf\" (UID: \"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" Nov 01 22:45:00.585259 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.585000 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6d8cdd-7079-4eb3-8701-d26549713a55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56\" (UID: \"5d6d8cdd-7079-4eb3-8701-d26549713a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" Nov 01 22:45:00.585259 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.585178 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9d81f5-1e85-41b2-9eac-c43ccfb98773-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf\" (UID: \"db9d81f5-1e85-41b2-9eac-c43ccfb98773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" Nov 01 22:45:00.585259 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.585253 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6gwm\" (UniqueName: \"kubernetes.io/projected/d680f9ed-27fd-44d0-a76c-1979761388a0-kube-api-access-j6gwm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn\" (UID: \"d680f9ed-27fd-44d0-a76c-1979761388a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" Nov 01 22:45:00.585428 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.585279 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d680f9ed-27fd-44d0-a76c-1979761388a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn\" (UID: \"d680f9ed-27fd-44d0-a76c-1979761388a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" Nov 01 22:45:00.585489 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.585439 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnfs2\" (UniqueName: \"kubernetes.io/projected/14785ed4-8e76-4509-9712-0026d2e80b74-kube-api-access-pnfs2\") pod \"maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j\" (UID: \"14785ed4-8e76-4509-9712-0026d2e80b74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" Nov 01 22:45:00.586563 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.586535 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14785ed4-8e76-4509-9712-0026d2e80b74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j\" (UID: \"14785ed4-8e76-4509-9712-0026d2e80b74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" Nov 01 22:45:00.587195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.587171 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9d81f5-1e85-41b2-9eac-c43ccfb98773-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf\" (UID: \"db9d81f5-1e85-41b2-9eac-c43ccfb98773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" Nov 01 22:45:00.587527 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.587505 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6d8cdd-7079-4eb3-8701-d26549713a55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56\" (UID: \"5d6d8cdd-7079-4eb3-8701-d26549713a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" Nov 01 22:45:00.595382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.595331 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnfs2\" (UniqueName: \"kubernetes.io/projected/14785ed4-8e76-4509-9712-0026d2e80b74-kube-api-access-pnfs2\") pod \"maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j\" (UID: \"14785ed4-8e76-4509-9712-0026d2e80b74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" Nov 01 22:45:00.599331 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.599134 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" Nov 01 22:45:00.599666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.599634 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqj6\" (UniqueName: \"kubernetes.io/projected/db9d81f5-1e85-41b2-9eac-c43ccfb98773-kube-api-access-vtqj6\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf\" (UID: \"db9d81f5-1e85-41b2-9eac-c43ccfb98773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" Nov 01 22:45:00.600828 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.600454 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9vgt\" (UniqueName: \"kubernetes.io/projected/5d6d8cdd-7079-4eb3-8701-d26549713a55-kube-api-access-v9vgt\") pod \"maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56\" (UID: \"5d6d8cdd-7079-4eb3-8701-d26549713a55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" Nov 01 22:45:00.639700 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.639465 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" Nov 01 22:45:00.649016 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.648987 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" Nov 01 22:45:00.690794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.689000 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6gwm\" (UniqueName: \"kubernetes.io/projected/d680f9ed-27fd-44d0-a76c-1979761388a0-kube-api-access-j6gwm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn\" (UID: \"d680f9ed-27fd-44d0-a76c-1979761388a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" Nov 01 22:45:00.690794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.689633 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d680f9ed-27fd-44d0-a76c-1979761388a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn\" (UID: \"d680f9ed-27fd-44d0-a76c-1979761388a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" Nov 01 22:45:00.697288 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.691021 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf\" (UID: \"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" Nov 01 22:45:00.697288 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.691725 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqkh\" (UniqueName: \"kubernetes.io/projected/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-kube-api-access-qqqkh\") pod \"maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf\" (UID: \"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" Nov 01 22:45:00.697288 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.696644 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d680f9ed-27fd-44d0-a76c-1979761388a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn\" (UID: \"d680f9ed-27fd-44d0-a76c-1979761388a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" Nov 01 22:45:00.703239 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.701810 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" Nov 01 22:45:00.712923 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.707645 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf\" (UID: \"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" Nov 01 22:45:00.729974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.727353 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqkh\" (UniqueName: \"kubernetes.io/projected/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-kube-api-access-qqqkh\") pod \"maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf\" (UID: \"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" Nov 01 22:45:00.738957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.738928 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6gwm\" (UniqueName: \"kubernetes.io/projected/d680f9ed-27fd-44d0-a76c-1979761388a0-kube-api-access-j6gwm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn\" (UID: \"d680f9ed-27fd-44d0-a76c-1979761388a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" Nov 01 22:45:00.765279 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.765028 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" Nov 01 22:45:00.792851 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.792620 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr"] Nov 01 22:45:00.793644 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:45:00.793617 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816a7bec_ff31_49a0_9b0c_0112e73b1a5a.slice/crio-1e14328d65a854aa06edcffae6b712d285b374b8514fe70466eac75343d2eac4 WatchSource:0}: Error finding container 1e14328d65a854aa06edcffae6b712d285b374b8514fe70466eac75343d2eac4: Status 404 returned error can't find the container with id 1e14328d65a854aa06edcffae6b712d285b374b8514fe70466eac75343d2eac4 Nov 01 22:45:00.834371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.831056 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" Nov 01 22:45:00.960401 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.960312 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" event={"ID":"816a7bec-ff31-49a0-9b0c-0112e73b1a5a","Type":"ContainerStarted","Data":"1e14328d65a854aa06edcffae6b712d285b374b8514fe70466eac75343d2eac4"} Nov 01 22:45:00.967475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.965929 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" event={"ID":"948c76bd-1965-496d-9c0d-cbb6bc453ffc","Type":"ContainerStarted","Data":"6a59eaf29dda5b74b21b0d6917afd14563d62b6b4655be6076ccb83da3b7cd4f"} Nov 01 22:45:00.967475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.965999 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" event={"ID":"948c76bd-1965-496d-9c0d-cbb6bc453ffc","Type":"ContainerStarted","Data":"55f33cc262977ed6ab82fe9ab408da9b3e1ec1ef9cd94fe1f10aa6753bed1e8c"} Nov 01 22:45:00.993302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:00.993200 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" podStartSLOduration=1.993178238 podStartE2EDuration="1.993178238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:00.988079289 +0000 UTC m=+1304.772518205" watchObservedRunningTime="2026-11-01 22:45:00.993178238 +0000 UTC m=+1304.777617160" Nov 01 22:45:01.285795 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.284963 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp"] Nov 01 22:45:01.289982 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.289891 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56"] Nov 01 22:45:01.294258 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:45:01.294232 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9d81f5_1e85_41b2_9eac_c43ccfb98773.slice/crio-0135c7544d62f3f67cb3cd89e30d3fc6df6ca4f5aba08f0d0b2d288f0c8447ff WatchSource:0}: Error finding container 0135c7544d62f3f67cb3cd89e30d3fc6df6ca4f5aba08f0d0b2d288f0c8447ff: Status 404 returned error can't find the container with id 0135c7544d62f3f67cb3cd89e30d3fc6df6ca4f5aba08f0d0b2d288f0c8447ff Nov 01 22:45:01.295979 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.295954 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j"] Nov 01 22:45:01.307333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.307143 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf"] Nov 01 22:45:01.418400 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:45:01.418368 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd680f9ed_27fd_44d0_a76c_1979761388a0.slice/crio-9e488d9c390b604f2be6595c6a0d8d665cdd4fef5d6e406e0f2fe21278aed792 WatchSource:0}: Error finding container 9e488d9c390b604f2be6595c6a0d8d665cdd4fef5d6e406e0f2fe21278aed792: Status 404 returned error can't find the container with id 9e488d9c390b604f2be6595c6a0d8d665cdd4fef5d6e406e0f2fe21278aed792 Nov 01 22:45:01.418730 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.418703 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn"] Nov 01 22:45:01.450276 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:45:01.450242 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7562a4b4_c136_4cd1_98c5_aa27d2ec9cb7.slice/crio-ad73ef8c046152a70fc025226363295adc662941a67a891b40c0e7db077f4e29 WatchSource:0}: Error finding container ad73ef8c046152a70fc025226363295adc662941a67a891b40c0e7db077f4e29: Status 404 returned error can't find the container with id ad73ef8c046152a70fc025226363295adc662941a67a891b40c0e7db077f4e29 Nov 01 22:45:01.451259 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.451233 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf"] Nov 01 22:45:01.781478 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.781434 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:45:01.807522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.807493 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:45:01.975367 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.975328 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" event={"ID":"5d6d8cdd-7079-4eb3-8701-d26549713a55","Type":"ContainerStarted","Data":"7e5f7854914a41a258ea95ac3a7a49e09cacdc6596bc304c2c1d168fc6228a7d"} Nov 01 22:45:01.976645 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.976618 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" event={"ID":"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7","Type":"ContainerStarted","Data":"ad73ef8c046152a70fc025226363295adc662941a67a891b40c0e7db077f4e29"} Nov 01 22:45:01.977804 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.977778 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" event={"ID":"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e","Type":"ContainerStarted","Data":"8e3480922386d4972d05e24bd9724ce3afff2d3258449e135fc5efede2865032"} Nov 01 22:45:01.979044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.979023 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" event={"ID":"db9d81f5-1e85-41b2-9eac-c43ccfb98773","Type":"ContainerStarted","Data":"0135c7544d62f3f67cb3cd89e30d3fc6df6ca4f5aba08f0d0b2d288f0c8447ff"} Nov 01 22:45:01.980360 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.980336 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" event={"ID":"d680f9ed-27fd-44d0-a76c-1979761388a0","Type":"ContainerStarted","Data":"9e488d9c390b604f2be6595c6a0d8d665cdd4fef5d6e406e0f2fe21278aed792"} Nov 01 22:45:01.981652 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.981631 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" event={"ID":"14785ed4-8e76-4509-9712-0026d2e80b74","Type":"ContainerStarted","Data":"6c71954b0487b9e5c3df56f658a5a7e1c3f38c27feedb6f7e200d3632fe44f16"} Nov 01 22:45:01.981847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.981831 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:45:01.982889 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:01.982871 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:45:03.364376 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.364344 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:45:03.405194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.403996 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-data\") pod \"f41e060e-e33f-478c-8019-ede84aeb4d13\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " Nov 01 22:45:03.405194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.404060 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-envoy\") pod \"f41e060e-e33f-478c-8019-ede84aeb4d13\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " Nov 01 22:45:03.405194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.404102 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-certs\") pod \"f41e060e-e33f-478c-8019-ede84aeb4d13\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " Nov 01 22:45:03.405194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.404149 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-kube-api-access-b7n7f\" (UniqueName: \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-kube-api-access-b7n7f\") pod \"f41e060e-e33f-478c-8019-ede84aeb4d13\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " Nov 01 22:45:03.405194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.404184 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-credential-socket\") pod \"f41e060e-e33f-478c-8019-ede84aeb4d13\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " Nov 01 22:45:03.405194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.404415 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41e060e-e33f-478c-8019-ede84aeb4d13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41e060e-e33f-478c-8019-ede84aeb4d13-istiod-ca-cert\") pod \"f41e060e-e33f-478c-8019-ede84aeb4d13\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " Nov 01 22:45:03.405194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.404460 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-podinfo\") pod \"f41e060e-e33f-478c-8019-ede84aeb4d13\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " Nov 01 22:45:03.405194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.404501 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-socket\") pod \"f41e060e-e33f-478c-8019-ede84aeb4d13\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " Nov 01 22:45:03.405194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.404529 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-token\" (UniqueName: \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-token\") pod \"f41e060e-e33f-478c-8019-ede84aeb4d13\" (UID: \"f41e060e-e33f-478c-8019-ede84aeb4d13\") " Nov 01 22:45:03.407505 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.406610 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-socket" pod "f41e060e-e33f-478c-8019-ede84aeb4d13" (UID: "f41e060e-e33f-478c-8019-ede84aeb4d13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:03.407505 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.406698 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-certs" pod "f41e060e-e33f-478c-8019-ede84aeb4d13" (UID: "f41e060e-e33f-478c-8019-ede84aeb4d13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:03.407505 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.407139 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-credential-socket" pod "f41e060e-e33f-478c-8019-ede84aeb4d13" (UID: "f41e060e-e33f-478c-8019-ede84aeb4d13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:03.407505 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.407278 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41e060e-e33f-478c-8019-ede84aeb4d13-istiod-ca-cert" pod "f41e060e-e33f-478c-8019-ede84aeb4d13" (UID: "f41e060e-e33f-478c-8019-ede84aeb4d13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:03.408257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.408211 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.408495 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.408236 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.408495 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.408441 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41e060e-e33f-478c-8019-ede84aeb4d13-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.408495 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.408459 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.408812 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.408638 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-data" pod "f41e060e-e33f-478c-8019-ede84aeb4d13" (UID: "f41e060e-e33f-478c-8019-ede84aeb4d13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:03.411790 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.411563 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-envoy" pod "f41e060e-e33f-478c-8019-ede84aeb4d13" (UID: "f41e060e-e33f-478c-8019-ede84aeb4d13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:03.411970 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.411937 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-podinfo" pod "f41e060e-e33f-478c-8019-ede84aeb4d13" (UID: "f41e060e-e33f-478c-8019-ede84aeb4d13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:03.412361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.412335 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-token" pod "f41e060e-e33f-478c-8019-ede84aeb4d13" (UID: "f41e060e-e33f-478c-8019-ede84aeb4d13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:03.433001 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.432698 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-kube-api-access-b7n7f" pod "f41e060e-e33f-478c-8019-ede84aeb4d13" (UID: "f41e060e-e33f-478c-8019-ede84aeb4d13"). InnerVolumeSpecName "kube-api-access-b7n7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:03.510588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.510534 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.510588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.510576 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.510588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.510590 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.510863 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.510603 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41e060e-e33f-478c-8019-ede84aeb4d13-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.510863 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:03.510616 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7n7f\" (UniqueName: \"kubernetes.io/projected/f41e060e-e33f-478c-8019-ede84aeb4d13-kube-api-access-b7n7f\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:04.059516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.059473 6466 generic.go:356] "Generic (PLEG): container finished" podID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerID="b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78" exitCode=0 Nov 01 22:45:04.060703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.060675 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" Nov 01 22:45:04.062773 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.062447 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" event={"ID":"f41e060e-e33f-478c-8019-ede84aeb4d13","Type":"ContainerDied","Data":"b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78"} Nov 01 22:45:04.062773 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.062513 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5" event={"ID":"f41e060e-e33f-478c-8019-ede84aeb4d13","Type":"ContainerDied","Data":"9f9b7934a41dc46c2b2f9464c6d3d27456acad3657a55e50f2ad09f8095ce827"} Nov 01 22:45:04.062773 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.062546 6466 scope.go:122] "RemoveContainer" containerID="b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78" Nov 01 22:45:04.097499 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.097341 6466 scope.go:122] "RemoveContainer" containerID="64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e" Nov 01 22:45:04.103949 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.103913 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5"] Nov 01 22:45:04.115024 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.113990 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9b7320-openshift-default-79bd9f87b5-9tpp5"] Nov 01 22:45:04.122255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.122228 6466 scope.go:122] "RemoveContainer" containerID="b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78" Nov 01 22:45:04.123008 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:04.122839 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78\": container with ID starting with b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78 not found: ID does not exist" containerID="b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78" Nov 01 22:45:04.123008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.122881 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78"} err="failed to get container status \"b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78\": rpc error: code = NotFound desc = could not find container \"b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78\": container with ID starting with b7df849ea216991f9122684127e3e69c4a84f1462a919018b1dbb23b94f55d78 not found: ID does not exist" Nov 01 22:45:04.123008 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.122909 6466 scope.go:122] "RemoveContainer" containerID="64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e" Nov 01 22:45:04.125599 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:04.125565 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e\": container with ID starting with 64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e not found: ID does not exist" containerID="64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e" Nov 01 22:45:04.125733 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.125609 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e"} err="failed to get container status \"64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e\": rpc error: code = NotFound desc = could not find container \"64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e\": container with ID starting with 64691ef3941e96b0042da21f564af5f7314d972ff0c22478eb2d53e8b2c5a53e not found: ID does not exist" Nov 01 22:45:04.368772 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.368672 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" path="/var/lib/kubelet/pods/f41e060e-e33f-478c-8019-ede84aeb4d13/volumes" Nov 01 22:45:04.779136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.779064 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz"] Nov 01 22:45:04.783574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.781396 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" Nov 01 22:45:04.783574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.781423 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" Nov 01 22:45:04.783574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.781465 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" Nov 01 22:45:04.783574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.781473 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" Nov 01 22:45:04.783574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.781707 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" Nov 01 22:45:04.783574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.781730 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41e060e-e33f-478c-8019-ede84aeb4d13" containerName="istio-proxy" Nov 01 22:45:04.790225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.789065 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:45:04.794413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.794386 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1fddbd\"" Nov 01 22:45:04.794413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.794402 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz"] Nov 01 22:45:04.843917 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.843868 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7"] Nov 01 22:45:04.847195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.845493 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqdg\" (UniqueName: \"kubernetes.io/projected/6f67dae0-ae80-4a49-81a1-b5bdab84b335-kube-api-access-4bqdg\") pod \"payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz\" (UID: \"6f67dae0-ae80-4a49-81a1-b5bdab84b335\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:45:04.847195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.845582 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f67dae0-ae80-4a49-81a1-b5bdab84b335-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz\" (UID: \"6f67dae0-ae80-4a49-81a1-b5bdab84b335\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:45:04.853132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.851209 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:45:04.871824 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.871792 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1fddbd-dockercfg-d8gpq\"" Nov 01 22:45:04.871824 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.871800 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7"] Nov 01 22:45:04.949418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.947506 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqdg\" (UniqueName: \"kubernetes.io/projected/6f67dae0-ae80-4a49-81a1-b5bdab84b335-kube-api-access-4bqdg\") pod \"payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz\" (UID: \"6f67dae0-ae80-4a49-81a1-b5bdab84b335\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:45:04.949418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.947589 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kpz\" (UniqueName: \"kubernetes.io/projected/34b4edc8-46f2-48a0-98a8-78445a731089-kube-api-access-55kpz\") pod \"payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7\" (UID: \"34b4edc8-46f2-48a0-98a8-78445a731089\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:45:04.949418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.947641 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f67dae0-ae80-4a49-81a1-b5bdab84b335-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz\" (UID: \"6f67dae0-ae80-4a49-81a1-b5bdab84b335\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:45:04.949418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.948079 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b4edc8-46f2-48a0-98a8-78445a731089-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7\" (UID: \"34b4edc8-46f2-48a0-98a8-78445a731089\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:45:04.949418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.949063 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f67dae0-ae80-4a49-81a1-b5bdab84b335-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz\" (UID: \"6f67dae0-ae80-4a49-81a1-b5bdab84b335\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:45:04.976127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:04.975060 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqdg\" (UniqueName: \"kubernetes.io/projected/6f67dae0-ae80-4a49-81a1-b5bdab84b335-kube-api-access-4bqdg\") pod \"payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz\" (UID: \"6f67dae0-ae80-4a49-81a1-b5bdab84b335\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:45:05.097179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:05.091707 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b4edc8-46f2-48a0-98a8-78445a731089-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7\" (UID: \"34b4edc8-46f2-48a0-98a8-78445a731089\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:45:05.097179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:05.091791 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kpz\" (UniqueName: \"kubernetes.io/projected/34b4edc8-46f2-48a0-98a8-78445a731089-kube-api-access-55kpz\") pod \"payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7\" (UID: \"34b4edc8-46f2-48a0-98a8-78445a731089\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:45:05.138156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:05.137187 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:45:05.249181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:05.216362 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b4edc8-46f2-48a0-98a8-78445a731089-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7\" (UID: \"34b4edc8-46f2-48a0-98a8-78445a731089\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:45:05.391210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:05.386336 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kpz\" (UniqueName: \"kubernetes.io/projected/34b4edc8-46f2-48a0-98a8-78445a731089-kube-api-access-55kpz\") pod \"payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7\" (UID: \"34b4edc8-46f2-48a0-98a8-78445a731089\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:45:05.498522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:05.498461 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:45:06.054147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:06.046552 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7"] Nov 01 22:45:06.089179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:06.089144 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz"] Nov 01 22:45:06.097340 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:45:06.097286 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f67dae0_ae80_4a49_81a1_b5bdab84b335.slice/crio-2924be275a52e8dd363e88a143c33dbc065b941dbf466b3b8b192108a6c5a982 WatchSource:0}: Error finding container 2924be275a52e8dd363e88a143c33dbc065b941dbf466b3b8b192108a6c5a982: Status 404 returned error can't find the container with id 2924be275a52e8dd363e88a143c33dbc065b941dbf466b3b8b192108a6c5a982 Nov 01 22:45:06.417661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:06.416395 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" event={"ID":"6f67dae0-ae80-4a49-81a1-b5bdab84b335","Type":"ContainerStarted","Data":"bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8"} Nov 01 22:45:06.417661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:06.416462 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" event={"ID":"6f67dae0-ae80-4a49-81a1-b5bdab84b335","Type":"ContainerStarted","Data":"2924be275a52e8dd363e88a143c33dbc065b941dbf466b3b8b192108a6c5a982"} Nov 01 22:45:06.417661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:06.417600 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:45:06.427180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:06.427100 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" event={"ID":"34b4edc8-46f2-48a0-98a8-78445a731089","Type":"ContainerStarted","Data":"f8959c18ce649425f1463b08ea63f66c23e866f0747a62e90c4114f44b5c019e"} Nov 01 22:45:06.427335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:06.427183 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" event={"ID":"34b4edc8-46f2-48a0-98a8-78445a731089","Type":"ContainerStarted","Data":"1d741e209a965cd701e69c988d7bc4bca1af086163a9ce6d0d82c4e6433b862f"} Nov 01 22:45:06.427525 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:06.427508 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:45:06.445989 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:06.445941 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" podStartSLOduration=2.445928331 podStartE2EDuration="2.445928331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:06.442078911 +0000 UTC m=+1310.226517834" watchObservedRunningTime="2026-11-01 22:45:06.445928331 +0000 UTC m=+1310.230367269" Nov 01 22:45:06.470296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:06.469828 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" podStartSLOduration=2.469809527 podStartE2EDuration="2.469809527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:06.467708764 +0000 UTC m=+1310.252147689" watchObservedRunningTime="2026-11-01 22:45:06.469809527 +0000 UTC m=+1310.254248450" Nov 01 22:45:08.566598 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:08.566567 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst"] Nov 01 22:45:08.567075 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:08.567061 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm"] Nov 01 22:45:08.567946 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:08.567929 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck"] Nov 01 22:45:08.568453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:08.568431 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" containerID="cri-o://3187ed6473fad3e6d71b6fcbf35ad085e783d2403da23bbaaf06cc95cc0915df" gracePeriod=30 Nov 01 22:45:08.568712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:08.568235 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" containerID="cri-o://20e87c629602f35e99ec417a568d9676750596ab9dbac61019bd4b23d0ea005f" gracePeriod=30 Nov 01 22:45:08.568865 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:08.567675 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerName="istio-proxy" containerID="cri-o://8d6985449d07ccb5ed805713cbcbb55ae07e7d56c2ab47af9dc8467cc30207e3" gracePeriod=30 Nov 01 22:45:12.801018 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.799278 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" event={"ID":"14785ed4-8e76-4509-9712-0026d2e80b74","Type":"ContainerStarted","Data":"097a0838283f3b1b7c23a526baf62e0732583208ed7c36eb14f755cc7b1425bd"} Nov 01 22:45:12.821326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.818788 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" event={"ID":"5d6d8cdd-7079-4eb3-8701-d26549713a55","Type":"ContainerStarted","Data":"e976f6ec1237ecbe4107bd15f00199f5c3c8f5e0850c7c32c0c840e745173b73"} Nov 01 22:45:12.840586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.838273 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" event={"ID":"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7","Type":"ContainerStarted","Data":"307eb9e9d06b47781b81ee3bb44e53179467445672142994e5381767657c5d45"} Nov 01 22:45:12.850951 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.842022 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" event={"ID":"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e","Type":"ContainerStarted","Data":"917439e2edb6f75e3e6e778fbd7af170ba2073599a870370fd5a712e56f0143a"} Nov 01 22:45:12.857607 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.857567 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" event={"ID":"db9d81f5-1e85-41b2-9eac-c43ccfb98773","Type":"ContainerStarted","Data":"4246e7d6de09f7147fd7dbc8ada26ab100ec907419c767d646814f5b99b11ef9"} Nov 01 22:45:12.857940 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.857917 6466 scope.go:122] "RemoveContainer" containerID="4246e7d6de09f7147fd7dbc8ada26ab100ec907419c767d646814f5b99b11ef9" Nov 01 22:45:12.865810 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.865780 6466 generic.go:356] "Generic (PLEG): container finished" podID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerID="fad4e6653a2a1fc53613bb5c37d15640e201ad05103114be9e425d6fba307030" exitCode=60 Nov 01 22:45:12.865900 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.865878 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" event={"ID":"d680f9ed-27fd-44d0-a76c-1979761388a0","Type":"ContainerDied","Data":"fad4e6653a2a1fc53613bb5c37d15640e201ad05103114be9e425d6fba307030"} Nov 01 22:45:12.867709 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.867684 6466 scope.go:122] "RemoveContainer" containerID="fad4e6653a2a1fc53613bb5c37d15640e201ad05103114be9e425d6fba307030" Nov 01 22:45:12.869336 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.869281 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" podStartSLOduration=2.722888904 podStartE2EDuration="12.869262492s" totalImagesPullingTime="10.146373588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:12.839984545 +0000 UTC m=+1316.624423468" watchObservedRunningTime="2026-11-01 22:45:12.869262492 +0000 UTC m=+1316.653701413" Nov 01 22:45:12.875922 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.875064 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" event={"ID":"816a7bec-ff31-49a0-9b0c-0112e73b1a5a","Type":"ContainerStarted","Data":"4322dbf7564468dcf5d13dd317cc1e8d060363cc77c5ce42b8e6587b55ec9ef6"} Nov 01 22:45:12.877292 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.877270 6466 scope.go:122] "RemoveContainer" containerID="4322dbf7564468dcf5d13dd317cc1e8d060363cc77c5ce42b8e6587b55ec9ef6" Nov 01 22:45:12.880654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.880610 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" podStartSLOduration=2.6602147560000002 podStartE2EDuration="12.880597246s" totalImagesPullingTime="10.22038249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:12.864814959 +0000 UTC m=+1316.649253880" watchObservedRunningTime="2026-11-01 22:45:12.880597246 +0000 UTC m=+1316.665036173" Nov 01 22:45:12.974776 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.974726 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" podStartSLOduration=2.82084839 podStartE2EDuration="12.974709931s" totalImagesPullingTime="10.153861541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:12.930307661 +0000 UTC m=+1316.714746585" watchObservedRunningTime="2026-11-01 22:45:12.974709931 +0000 UTC m=+1316.759148917" Nov 01 22:45:12.975693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:12.975649 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" podStartSLOduration=2.753487689 podStartE2EDuration="12.975639359s" totalImagesPullingTime="10.22215167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:12.970476876 +0000 UTC m=+1316.754916016" watchObservedRunningTime="2026-11-01 22:45:12.975639359 +0000 UTC m=+1316.760078284" Nov 01 22:45:13.894815 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.894714 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" event={"ID":"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2","Type":"ContainerDied","Data":"3187ed6473fad3e6d71b6fcbf35ad085e783d2403da23bbaaf06cc95cc0915df"} Nov 01 22:45:13.895409 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.895023 6466 generic.go:356] "Generic (PLEG): container finished" podID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerID="3187ed6473fad3e6d71b6fcbf35ad085e783d2403da23bbaaf06cc95cc0915df" exitCode=0 Nov 01 22:45:13.899490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.898433 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" event={"ID":"d680f9ed-27fd-44d0-a76c-1979761388a0","Type":"ContainerStarted","Data":"c06c77feb7d123676b8e158be47c0c770e8f89768feb1919aaf2f4a8ac90d3d6"} Nov 01 22:45:13.902157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.901249 6466 generic.go:356] "Generic (PLEG): container finished" podID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerID="4322dbf7564468dcf5d13dd317cc1e8d060363cc77c5ce42b8e6587b55ec9ef6" exitCode=60 Nov 01 22:45:13.902157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.901404 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" event={"ID":"816a7bec-ff31-49a0-9b0c-0112e73b1a5a","Type":"ContainerDied","Data":"4322dbf7564468dcf5d13dd317cc1e8d060363cc77c5ce42b8e6587b55ec9ef6"} Nov 01 22:45:13.902157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.901434 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" event={"ID":"816a7bec-ff31-49a0-9b0c-0112e73b1a5a","Type":"ContainerStarted","Data":"d70002ea99a4945fd8b35457368ea33410294267f7698ee5a275ba71dadfb4db"} Nov 01 22:45:13.903720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.903613 6466 generic.go:356] "Generic (PLEG): container finished" podID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerID="8d6985449d07ccb5ed805713cbcbb55ae07e7d56c2ab47af9dc8467cc30207e3" exitCode=0 Nov 01 22:45:13.903720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.903671 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" event={"ID":"42a544a3-27d3-43ff-98c4-b47975cfc225","Type":"ContainerDied","Data":"8d6985449d07ccb5ed805713cbcbb55ae07e7d56c2ab47af9dc8467cc30207e3"} Nov 01 22:45:13.906524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.905474 6466 generic.go:356] "Generic (PLEG): container finished" podID="14785ed4-8e76-4509-9712-0026d2e80b74" containerID="097a0838283f3b1b7c23a526baf62e0732583208ed7c36eb14f755cc7b1425bd" exitCode=60 Nov 01 22:45:13.906524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.905572 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" event={"ID":"14785ed4-8e76-4509-9712-0026d2e80b74","Type":"ContainerDied","Data":"097a0838283f3b1b7c23a526baf62e0732583208ed7c36eb14f755cc7b1425bd"} Nov 01 22:45:13.906524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.905894 6466 scope.go:122] "RemoveContainer" containerID="097a0838283f3b1b7c23a526baf62e0732583208ed7c36eb14f755cc7b1425bd" Nov 01 22:45:13.933277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.931424 6466 generic.go:356] "Generic (PLEG): container finished" podID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerID="e976f6ec1237ecbe4107bd15f00199f5c3c8f5e0850c7c32c0c840e745173b73" exitCode=60 Nov 01 22:45:13.933277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.931826 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" event={"ID":"5d6d8cdd-7079-4eb3-8701-d26549713a55","Type":"ContainerDied","Data":"e976f6ec1237ecbe4107bd15f00199f5c3c8f5e0850c7c32c0c840e745173b73"} Nov 01 22:45:13.959131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.954183 6466 scope.go:122] "RemoveContainer" containerID="e976f6ec1237ecbe4107bd15f00199f5c3c8f5e0850c7c32c0c840e745173b73" Nov 01 22:45:13.982858 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.982791 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" podStartSLOduration=4.195814466 podStartE2EDuration="13.982769552s" totalImagesPullingTime="9.786955086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:13.977923331 +0000 UTC m=+1317.762362253" watchObservedRunningTime="2026-11-01 22:45:13.982769552 +0000 UTC m=+1317.767208472" Nov 01 22:45:13.990637 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.990602 6466 generic.go:356] "Generic (PLEG): container finished" podID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerID="20e87c629602f35e99ec417a568d9676750596ab9dbac61019bd4b23d0ea005f" exitCode=0 Nov 01 22:45:13.990837 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.990710 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" event={"ID":"ff42c4d5-6fe0-4a47-b371-238761f656dc","Type":"ContainerDied","Data":"20e87c629602f35e99ec417a568d9676750596ab9dbac61019bd4b23d0ea005f"} Nov 01 22:45:13.994369 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.994335 6466 generic.go:356] "Generic (PLEG): container finished" podID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerID="307eb9e9d06b47781b81ee3bb44e53179467445672142994e5381767657c5d45" exitCode=60 Nov 01 22:45:13.994498 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.994425 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" event={"ID":"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7","Type":"ContainerDied","Data":"307eb9e9d06b47781b81ee3bb44e53179467445672142994e5381767657c5d45"} Nov 01 22:45:13.994988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.994965 6466 scope.go:122] "RemoveContainer" containerID="307eb9e9d06b47781b81ee3bb44e53179467445672142994e5381767657c5d45" Nov 01 22:45:13.995492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:13.995465 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:45:14.000595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.000569 6466 generic.go:356] "Generic (PLEG): container finished" podID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerID="917439e2edb6f75e3e6e778fbd7af170ba2073599a870370fd5a712e56f0143a" exitCode=60 Nov 01 22:45:14.000727 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.000707 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" event={"ID":"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e","Type":"ContainerDied","Data":"917439e2edb6f75e3e6e778fbd7af170ba2073599a870370fd5a712e56f0143a"} Nov 01 22:45:14.001232 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.001207 6466 scope.go:122] "RemoveContainer" containerID="917439e2edb6f75e3e6e778fbd7af170ba2073599a870370fd5a712e56f0143a" Nov 01 22:45:14.005190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.005162 6466 generic.go:356] "Generic (PLEG): container finished" podID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerID="4246e7d6de09f7147fd7dbc8ada26ab100ec907419c767d646814f5b99b11ef9" exitCode=60 Nov 01 22:45:14.005293 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.005275 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" event={"ID":"db9d81f5-1e85-41b2-9eac-c43ccfb98773","Type":"ContainerDied","Data":"4246e7d6de09f7147fd7dbc8ada26ab100ec907419c767d646814f5b99b11ef9"} Nov 01 22:45:14.005353 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.005318 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" event={"ID":"db9d81f5-1e85-41b2-9eac-c43ccfb98773","Type":"ContainerStarted","Data":"06592c7757bec3cdce05c5bb910bc196a35306c417392ef1c99234aeeac757dd"} Nov 01 22:45:14.031648 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.031596 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" podStartSLOduration=3.497138515 podStartE2EDuration="14.031577642s" totalImagesPullingTime="10.534439127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:14.029880367 +0000 UTC m=+1317.814319288" watchObservedRunningTime="2026-11-01 22:45:14.031577642 +0000 UTC m=+1317.816016564" Nov 01 22:45:14.116686 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.116642 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-kube-api-access-q6496\" (UniqueName: \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-kube-api-access-q6496\") pod \"42a544a3-27d3-43ff-98c4-b47975cfc225\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " Nov 01 22:45:14.117466 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.117440 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-credential-socket\") pod \"42a544a3-27d3-43ff-98c4-b47975cfc225\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " Nov 01 22:45:14.118084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.117880 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-credential-socket" pod "42a544a3-27d3-43ff-98c4-b47975cfc225" (UID: "42a544a3-27d3-43ff-98c4-b47975cfc225"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.118253 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.118236 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a544a3-27d3-43ff-98c4-b47975cfc225-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a544a3-27d3-43ff-98c4-b47975cfc225-istiod-ca-cert\") pod \"42a544a3-27d3-43ff-98c4-b47975cfc225\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " Nov 01 22:45:14.118797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.118691 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a544a3-27d3-43ff-98c4-b47975cfc225-istiod-ca-cert" pod "42a544a3-27d3-43ff-98c4-b47975cfc225" (UID: "42a544a3-27d3-43ff-98c4-b47975cfc225"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:14.119232 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.119215 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-token\" (UniqueName: \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-token\") pod \"42a544a3-27d3-43ff-98c4-b47975cfc225\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " Nov 01 22:45:14.119442 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.119424 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-podinfo\") pod \"42a544a3-27d3-43ff-98c4-b47975cfc225\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " Nov 01 22:45:14.119904 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.119887 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-certs\") pod \"42a544a3-27d3-43ff-98c4-b47975cfc225\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " Nov 01 22:45:14.120448 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.120343 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-certs" pod "42a544a3-27d3-43ff-98c4-b47975cfc225" (UID: "42a544a3-27d3-43ff-98c4-b47975cfc225"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.120833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.120813 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-socket" pod "42a544a3-27d3-43ff-98c4-b47975cfc225" (UID: "42a544a3-27d3-43ff-98c4-b47975cfc225"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.121120 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.121088 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-socket\") pod \"42a544a3-27d3-43ff-98c4-b47975cfc225\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " Nov 01 22:45:14.121292 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.121275 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-envoy\") pod \"42a544a3-27d3-43ff-98c4-b47975cfc225\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " Nov 01 22:45:14.122136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.122089 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-data" pod "42a544a3-27d3-43ff-98c4-b47975cfc225" (UID: "42a544a3-27d3-43ff-98c4-b47975cfc225"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.122424 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.122408 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-data\") pod \"42a544a3-27d3-43ff-98c4-b47975cfc225\" (UID: \"42a544a3-27d3-43ff-98c4-b47975cfc225\") " Nov 01 22:45:14.127097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.127074 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.127270 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.127256 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.127379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.127365 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a544a3-27d3-43ff-98c4-b47975cfc225-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.127472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.127459 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.127551 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.127540 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.176270 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.175632 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" podStartSLOduration=4.275603727 podStartE2EDuration="14.175507302s" totalImagesPullingTime="9.899903575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:14.128651913 +0000 UTC m=+1317.913090835" watchObservedRunningTime="2026-11-01 22:45:14.175507302 +0000 UTC m=+1317.959946224" Nov 01 22:45:14.184585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.184504 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-podinfo" pod "42a544a3-27d3-43ff-98c4-b47975cfc225" (UID: "42a544a3-27d3-43ff-98c4-b47975cfc225"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:14.185083 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.184921 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-token" pod "42a544a3-27d3-43ff-98c4-b47975cfc225" (UID: "42a544a3-27d3-43ff-98c4-b47975cfc225"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:14.185083 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.184966 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-kube-api-access-q6496" pod "42a544a3-27d3-43ff-98c4-b47975cfc225" (UID: "42a544a3-27d3-43ff-98c4-b47975cfc225"). InnerVolumeSpecName "kube-api-access-q6496". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:14.185083 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.185032 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-envoy" pod "42a544a3-27d3-43ff-98c4-b47975cfc225" (UID: "42a544a3-27d3-43ff-98c4-b47975cfc225"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.321538 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.318162 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6496\" (UniqueName: \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-kube-api-access-q6496\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.321538 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.318189 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.321538 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.318202 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.321538 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.318216 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a544a3-27d3-43ff-98c4-b47975cfc225-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.484478 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.484447 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:45:14.521638 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.521609 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:45:14.522174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.521657 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-envoy\") pod \"ff42c4d5-6fe0-4a47-b371-238761f656dc\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " Nov 01 22:45:14.522505 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.522486 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-credential-socket\") pod \"ff42c4d5-6fe0-4a47-b371-238761f656dc\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " Nov 01 22:45:14.522618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.522605 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-kube-api-access-cwlcf\" (UniqueName: \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-kube-api-access-cwlcf\") pod \"ff42c4d5-6fe0-4a47-b371-238761f656dc\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " Nov 01 22:45:14.522793 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.522721 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-token\" (UniqueName: \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-token\") pod \"ff42c4d5-6fe0-4a47-b371-238761f656dc\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " Nov 01 22:45:14.522922 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.522908 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-data\") pod \"ff42c4d5-6fe0-4a47-b371-238761f656dc\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " Nov 01 22:45:14.523219 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.523201 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff42c4d5-6fe0-4a47-b371-238761f656dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff42c4d5-6fe0-4a47-b371-238761f656dc-istiod-ca-cert\") pod \"ff42c4d5-6fe0-4a47-b371-238761f656dc\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " Nov 01 22:45:14.523452 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.523438 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-socket\") pod \"ff42c4d5-6fe0-4a47-b371-238761f656dc\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " Nov 01 22:45:14.524175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.524155 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-podinfo\") pod \"ff42c4d5-6fe0-4a47-b371-238761f656dc\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " Nov 01 22:45:14.524666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.524646 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-certs\") pod \"ff42c4d5-6fe0-4a47-b371-238761f656dc\" (UID: \"ff42c4d5-6fe0-4a47-b371-238761f656dc\") " Nov 01 22:45:14.525429 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.522810 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-credential-socket" pod "ff42c4d5-6fe0-4a47-b371-238761f656dc" (UID: "ff42c4d5-6fe0-4a47-b371-238761f656dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.525565 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.523761 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-socket" pod "ff42c4d5-6fe0-4a47-b371-238761f656dc" (UID: "ff42c4d5-6fe0-4a47-b371-238761f656dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.525704 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.524015 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff42c4d5-6fe0-4a47-b371-238761f656dc-istiod-ca-cert" pod "ff42c4d5-6fe0-4a47-b371-238761f656dc" (UID: "ff42c4d5-6fe0-4a47-b371-238761f656dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:14.525832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.524059 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-data" pod "ff42c4d5-6fe0-4a47-b371-238761f656dc" (UID: "ff42c4d5-6fe0-4a47-b371-238761f656dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.525947 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.525199 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-certs" pod "ff42c4d5-6fe0-4a47-b371-238761f656dc" (UID: "ff42c4d5-6fe0-4a47-b371-238761f656dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.528781 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.528749 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.528781 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.528775 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.528925 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.528790 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff42c4d5-6fe0-4a47-b371-238761f656dc-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.528925 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.528804 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.528925 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.528819 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.529534 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.529508 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-podinfo" pod "ff42c4d5-6fe0-4a47-b371-238761f656dc" (UID: "ff42c4d5-6fe0-4a47-b371-238761f656dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:14.530543 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.530520 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-envoy" pod "ff42c4d5-6fe0-4a47-b371-238761f656dc" (UID: "ff42c4d5-6fe0-4a47-b371-238761f656dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.576048 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.575313 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-kube-api-access-cwlcf" pod "ff42c4d5-6fe0-4a47-b371-238761f656dc" (UID: "ff42c4d5-6fe0-4a47-b371-238761f656dc"). InnerVolumeSpecName "kube-api-access-cwlcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:14.576048 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.575915 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-token" pod "ff42c4d5-6fe0-4a47-b371-238761f656dc" (UID: "ff42c4d5-6fe0-4a47-b371-238761f656dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:14.630037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.629944 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istiod-ca-cert\") pod \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " Nov 01 22:45:14.630937 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.630898 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-certs" pod "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" (UID: "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.631374 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.631343 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istiod-ca-cert" pod "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" (UID: "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:14.631474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.631417 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-certs\") pod \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " Nov 01 22:45:14.631474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.631461 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-credential-socket\") pod \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " Nov 01 22:45:14.631586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.631505 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-podinfo\") pod \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " Nov 01 22:45:14.631586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.631548 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-kube-api-access-vp8rx\" (UniqueName: \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-kube-api-access-vp8rx\") pod \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " Nov 01 22:45:14.631712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.631612 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-data\") pod \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " Nov 01 22:45:14.631766 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.631753 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-token\" (UniqueName: \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-token\") pod \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " Nov 01 22:45:14.631857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.631839 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-envoy\") pod \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " Nov 01 22:45:14.631949 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.631932 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-socket\") pod \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\" (UID: \"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2\") " Nov 01 22:45:14.633003 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.632978 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.633126 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.633009 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.633126 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.633024 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwlcf\" (UniqueName: \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-kube-api-access-cwlcf\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.633126 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.633036 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff42c4d5-6fe0-4a47-b371-238761f656dc-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.633126 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.633050 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.633126 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.633065 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.633553 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.633526 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-credential-socket" pod "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" (UID: "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.633744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.633718 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-socket" pod "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" (UID: "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.633888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.633846 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-data" pod "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" (UID: "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.667455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.667334 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-kube-api-access-vp8rx" pod "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" (UID: "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2"). InnerVolumeSpecName "kube-api-access-vp8rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:14.668732 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.668659 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-envoy" pod "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" (UID: "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.670209 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.670184 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-token" pod "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" (UID: "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:14.680626 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.680575 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-podinfo" pod "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" (UID: "e5c86719-9d61-4c99-b28f-6eba7ebe6ae2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:14.734625 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.734598 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.734625 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.734624 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.734849 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.734635 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp8rx\" (UniqueName: \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-kube-api-access-vp8rx\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.734849 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.734645 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.734849 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.734654 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.734849 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.734663 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.734849 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:14.734674 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:15.016814 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.016780 6466 generic.go:356] "Generic (PLEG): container finished" podID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerID="06592c7757bec3cdce05c5bb910bc196a35306c417392ef1c99234aeeac757dd" exitCode=60 Nov 01 22:45:15.017253 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.016864 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" event={"ID":"db9d81f5-1e85-41b2-9eac-c43ccfb98773","Type":"ContainerDied","Data":"06592c7757bec3cdce05c5bb910bc196a35306c417392ef1c99234aeeac757dd"} Nov 01 22:45:15.017253 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.016926 6466 scope.go:122] "RemoveContainer" containerID="4246e7d6de09f7147fd7dbc8ada26ab100ec907419c767d646814f5b99b11ef9" Nov 01 22:45:15.017388 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.017251 6466 scope.go:122] "RemoveContainer" containerID="06592c7757bec3cdce05c5bb910bc196a35306c417392ef1c99234aeeac757dd" Nov 01 22:45:15.017689 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:15.017661 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf_odh-ai-gateway-infra(db9d81f5-1e85-41b2-9eac-c43ccfb98773)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" Nov 01 22:45:15.018562 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.018535 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" Nov 01 22:45:15.018652 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.018575 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck" event={"ID":"e5c86719-9d61-4c99-b28f-6eba7ebe6ae2","Type":"ContainerDied","Data":"9a89e80a4949f6dd0b5946cf8a28f52c632bcbbe9b23e87722eef942f03a4664"} Nov 01 22:45:15.021236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.021210 6466 generic.go:356] "Generic (PLEG): container finished" podID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerID="c06c77feb7d123676b8e158be47c0c770e8f89768feb1919aaf2f4a8ac90d3d6" exitCode=60 Nov 01 22:45:15.021329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.021275 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" event={"ID":"d680f9ed-27fd-44d0-a76c-1979761388a0","Type":"ContainerDied","Data":"c06c77feb7d123676b8e158be47c0c770e8f89768feb1919aaf2f4a8ac90d3d6"} Nov 01 22:45:15.021734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.021655 6466 scope.go:122] "RemoveContainer" containerID="c06c77feb7d123676b8e158be47c0c770e8f89768feb1919aaf2f4a8ac90d3d6" Nov 01 22:45:15.021989 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:15.021963 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn_odh-ai-gateway-infra(d680f9ed-27fd-44d0-a76c-1979761388a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" Nov 01 22:45:15.023819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.023800 6466 generic.go:356] "Generic (PLEG): container finished" podID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerID="d70002ea99a4945fd8b35457368ea33410294267f7698ee5a275ba71dadfb4db" exitCode=60 Nov 01 22:45:15.023964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.023950 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" event={"ID":"816a7bec-ff31-49a0-9b0c-0112e73b1a5a","Type":"ContainerDied","Data":"d70002ea99a4945fd8b35457368ea33410294267f7698ee5a275ba71dadfb4db"} Nov 01 22:45:15.024456 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.024330 6466 scope.go:122] "RemoveContainer" containerID="d70002ea99a4945fd8b35457368ea33410294267f7698ee5a275ba71dadfb4db" Nov 01 22:45:15.024636 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:15.024615 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr_odh-ai-gateway-infra(816a7bec-ff31-49a0-9b0c-0112e73b1a5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" Nov 01 22:45:15.025741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.025719 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" Nov 01 22:45:15.025741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.025726 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst" event={"ID":"42a544a3-27d3-43ff-98c4-b47975cfc225","Type":"ContainerDied","Data":"22ae6dc91cc8dd6c417eeb09757646bbd516d43cb7336cccf89481bcf8c01c2c"} Nov 01 22:45:15.028580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.028513 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" event={"ID":"14785ed4-8e76-4509-9712-0026d2e80b74","Type":"ContainerStarted","Data":"33497e6901ac89a9904f8a11e574694a2df9fc4fddf7b3f5b93868159acca2d3"} Nov 01 22:45:15.031830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.031807 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" event={"ID":"5d6d8cdd-7079-4eb3-8701-d26549713a55","Type":"ContainerStarted","Data":"9ddc3b9f300988db7ddd95653fcae57c8ec5f11907c024b734f18c3cd939babb"} Nov 01 22:45:15.068332 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.068294 6466 scope.go:122] "RemoveContainer" containerID="3187ed6473fad3e6d71b6fcbf35ad085e783d2403da23bbaaf06cc95cc0915df" Nov 01 22:45:15.068673 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.068650 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" event={"ID":"ff42c4d5-6fe0-4a47-b371-238761f656dc","Type":"ContainerDied","Data":"5fb002057170edac3a2e6ba0e2f488e083123289d28549a9b1261cf7c31febe0"} Nov 01 22:45:15.068756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.068747 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" Nov 01 22:45:15.073240 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.073205 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" event={"ID":"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7","Type":"ContainerStarted","Data":"e7cd3bd64260a4d5f97ec73b8d05bce4b24167b930123a97f174f5eed77d9f5f"} Nov 01 22:45:15.075697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.075668 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" event={"ID":"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e","Type":"ContainerStarted","Data":"37d1a84ed3dd4fcb56315922ab5d9269b698349c4088ad1f459fee594932f168"} Nov 01 22:45:15.082351 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.082332 6466 scope.go:122] "RemoveContainer" containerID="fad4e6653a2a1fc53613bb5c37d15640e201ad05103114be9e425d6fba307030" Nov 01 22:45:15.097834 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.097816 6466 scope.go:122] "RemoveContainer" containerID="4322dbf7564468dcf5d13dd317cc1e8d060363cc77c5ce42b8e6587b55ec9ef6" Nov 01 22:45:15.112291 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.112259 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck"] Nov 01 22:45:15.114600 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.114580 6466 scope.go:122] "RemoveContainer" containerID="8d6985449d07ccb5ed805713cbcbb55ae07e7d56c2ab47af9dc8467cc30207e3" Nov 01 22:45:15.117374 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.117351 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-94qck"] Nov 01 22:45:15.126266 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.126241 6466 scope.go:122] "RemoveContainer" containerID="20e87c629602f35e99ec417a568d9676750596ab9dbac61019bd4b23d0ea005f" Nov 01 22:45:15.167193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.166951 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst"] Nov 01 22:45:15.171583 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.171555 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-h2nst"] Nov 01 22:45:15.257585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.257203 6466 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:45:15.257585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.257260 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:15.457142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.435989 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm"] Nov 01 22:45:15.484510 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:15.484480 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-sm4gm"] Nov 01 22:45:16.116333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.116242 6466 scope.go:122] "RemoveContainer" containerID="c06c77feb7d123676b8e158be47c0c770e8f89768feb1919aaf2f4a8ac90d3d6" Nov 01 22:45:16.117557 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:16.117525 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn_odh-ai-gateway-infra(d680f9ed-27fd-44d0-a76c-1979761388a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" Nov 01 22:45:16.141960 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.141916 6466 scope.go:122] "RemoveContainer" containerID="d70002ea99a4945fd8b35457368ea33410294267f7698ee5a275ba71dadfb4db" Nov 01 22:45:16.142721 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:16.142692 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr_odh-ai-gateway-infra(816a7bec-ff31-49a0-9b0c-0112e73b1a5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" Nov 01 22:45:16.212838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.212803 6466 generic.go:356] "Generic (PLEG): container finished" podID="14785ed4-8e76-4509-9712-0026d2e80b74" containerID="33497e6901ac89a9904f8a11e574694a2df9fc4fddf7b3f5b93868159acca2d3" exitCode=60 Nov 01 22:45:16.213194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.213165 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" event={"ID":"14785ed4-8e76-4509-9712-0026d2e80b74","Type":"ContainerDied","Data":"33497e6901ac89a9904f8a11e574694a2df9fc4fddf7b3f5b93868159acca2d3"} Nov 01 22:45:16.213412 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.213397 6466 scope.go:122] "RemoveContainer" containerID="097a0838283f3b1b7c23a526baf62e0732583208ed7c36eb14f755cc7b1425bd" Nov 01 22:45:16.214027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.214010 6466 scope.go:122] "RemoveContainer" containerID="33497e6901ac89a9904f8a11e574694a2df9fc4fddf7b3f5b93868159acca2d3" Nov 01 22:45:16.214681 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:16.214656 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j_odh-ai-gateway-infra(14785ed4-8e76-4509-9712-0026d2e80b74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" Nov 01 22:45:16.220183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.220160 6466 generic.go:356] "Generic (PLEG): container finished" podID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerID="9ddc3b9f300988db7ddd95653fcae57c8ec5f11907c024b734f18c3cd939babb" exitCode=60 Nov 01 22:45:16.220362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.220343 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" event={"ID":"5d6d8cdd-7079-4eb3-8701-d26549713a55","Type":"ContainerDied","Data":"9ddc3b9f300988db7ddd95653fcae57c8ec5f11907c024b734f18c3cd939babb"} Nov 01 22:45:16.222712 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.222687 6466 scope.go:122] "RemoveContainer" containerID="9ddc3b9f300988db7ddd95653fcae57c8ec5f11907c024b734f18c3cd939babb" Nov 01 22:45:16.223410 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:16.223387 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56_odh-ai-gateway-infra(5d6d8cdd-7079-4eb3-8701-d26549713a55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" Nov 01 22:45:16.224098 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.224071 6466 generic.go:356] "Generic (PLEG): container finished" podID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerID="e7cd3bd64260a4d5f97ec73b8d05bce4b24167b930123a97f174f5eed77d9f5f" exitCode=60 Nov 01 22:45:16.224209 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.224131 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" event={"ID":"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7","Type":"ContainerDied","Data":"e7cd3bd64260a4d5f97ec73b8d05bce4b24167b930123a97f174f5eed77d9f5f"} Nov 01 22:45:16.224616 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.224586 6466 scope.go:122] "RemoveContainer" containerID="e7cd3bd64260a4d5f97ec73b8d05bce4b24167b930123a97f174f5eed77d9f5f" Nov 01 22:45:16.225089 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:16.225056 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf_odh-ai-gateway-infra(7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" Nov 01 22:45:16.226352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.226333 6466 generic.go:356] "Generic (PLEG): container finished" podID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerID="37d1a84ed3dd4fcb56315922ab5d9269b698349c4088ad1f459fee594932f168" exitCode=60 Nov 01 22:45:16.226541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.226360 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" event={"ID":"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e","Type":"ContainerDied","Data":"37d1a84ed3dd4fcb56315922ab5d9269b698349c4088ad1f459fee594932f168"} Nov 01 22:45:16.226671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.226653 6466 scope.go:122] "RemoveContainer" containerID="37d1a84ed3dd4fcb56315922ab5d9269b698349c4088ad1f459fee594932f168" Nov 01 22:45:16.227032 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:16.227006 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp_odh-ai-gateway-infra(1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" Nov 01 22:45:16.229841 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.228877 6466 scope.go:122] "RemoveContainer" containerID="06592c7757bec3cdce05c5bb910bc196a35306c417392ef1c99234aeeac757dd" Nov 01 22:45:16.229841 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:16.229220 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf_odh-ai-gateway-infra(db9d81f5-1e85-41b2-9eac-c43ccfb98773)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" Nov 01 22:45:16.232455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.232439 6466 scope.go:122] "RemoveContainer" containerID="e976f6ec1237ecbe4107bd15f00199f5c3c8f5e0850c7c32c0c840e745173b73" Nov 01 22:45:16.281892 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.281870 6466 scope.go:122] "RemoveContainer" containerID="307eb9e9d06b47781b81ee3bb44e53179467445672142994e5381767657c5d45" Nov 01 22:45:16.310738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.310717 6466 scope.go:122] "RemoveContainer" containerID="917439e2edb6f75e3e6e778fbd7af170ba2073599a870370fd5a712e56f0143a" Nov 01 22:45:16.379401 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.379321 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" path="/var/lib/kubelet/pods/42a544a3-27d3-43ff-98c4-b47975cfc225/volumes" Nov 01 22:45:16.379942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.379926 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" path="/var/lib/kubelet/pods/e5c86719-9d61-4c99-b28f-6eba7ebe6ae2/volumes" Nov 01 22:45:16.380511 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:16.380495 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" path="/var/lib/kubelet/pods/ff42c4d5-6fe0-4a47-b371-238761f656dc/volumes" Nov 01 22:45:17.269493 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:17.269460 6466 scope.go:122] "RemoveContainer" containerID="33497e6901ac89a9904f8a11e574694a2df9fc4fddf7b3f5b93868159acca2d3" Nov 01 22:45:17.269962 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:17.269913 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j_odh-ai-gateway-infra(14785ed4-8e76-4509-9712-0026d2e80b74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" Nov 01 22:45:17.271608 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:17.271587 6466 scope.go:122] "RemoveContainer" containerID="9ddc3b9f300988db7ddd95653fcae57c8ec5f11907c024b734f18c3cd939babb" Nov 01 22:45:17.271964 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:17.271940 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56_odh-ai-gateway-infra(5d6d8cdd-7079-4eb3-8701-d26549713a55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" Nov 01 22:45:17.273329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:17.273313 6466 scope.go:122] "RemoveContainer" containerID="e7cd3bd64260a4d5f97ec73b8d05bce4b24167b930123a97f174f5eed77d9f5f" Nov 01 22:45:17.273656 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:17.273639 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf_odh-ai-gateway-infra(7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" Nov 01 22:45:17.274935 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:17.274902 6466 scope.go:122] "RemoveContainer" containerID="37d1a84ed3dd4fcb56315922ab5d9269b698349c4088ad1f459fee594932f168" Nov 01 22:45:17.275213 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:17.275182 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp_odh-ai-gateway-infra(1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" Nov 01 22:45:17.478980 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:17.478946 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:45:18.475557 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:18.475524 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g"] Nov 01 22:45:18.476172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:18.476144 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" containerID="cri-o://b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603" gracePeriod=30 Nov 01 22:45:18.563658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:18.563628 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:45:19.412411 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:19.412374 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:45:23.771079 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.771045 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:45:23.827760 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.827724 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e34de96-6bf5-411e-b72d-2010871edf08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e34de96-6bf5-411e-b72d-2010871edf08-istio-podinfo\") pod \"3e34de96-6bf5-411e-b72d-2010871edf08\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " Nov 01 22:45:23.827941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.827811 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-istio-token\" (UniqueName: \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-istio-token\") pod \"3e34de96-6bf5-411e-b72d-2010871edf08\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " Nov 01 22:45:23.827941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.827903 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-socket\") pod \"3e34de96-6bf5-411e-b72d-2010871edf08\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " Nov 01 22:45:23.828148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.827937 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-certs\") pod \"3e34de96-6bf5-411e-b72d-2010871edf08\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " Nov 01 22:45:23.828148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.828089 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-data\") pod \"3e34de96-6bf5-411e-b72d-2010871edf08\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " Nov 01 22:45:23.828488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.828254 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-socket" pod "3e34de96-6bf5-411e-b72d-2010871edf08" (UID: "3e34de96-6bf5-411e-b72d-2010871edf08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:23.828488 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.828478 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-envoy\") pod \"3e34de96-6bf5-411e-b72d-2010871edf08\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " Nov 01 22:45:23.828695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.828271 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-certs" pod "3e34de96-6bf5-411e-b72d-2010871edf08" (UID: "3e34de96-6bf5-411e-b72d-2010871edf08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:23.828695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.828363 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-data" pod "3e34de96-6bf5-411e-b72d-2010871edf08" (UID: "3e34de96-6bf5-411e-b72d-2010871edf08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:23.828695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.828609 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-kube-api-access-ltjdx\" (UniqueName: \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-kube-api-access-ltjdx\") pod \"3e34de96-6bf5-411e-b72d-2010871edf08\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " Nov 01 22:45:23.828695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.828667 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-credential-socket\") pod \"3e34de96-6bf5-411e-b72d-2010871edf08\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " Nov 01 22:45:23.828933 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.828803 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e34de96-6bf5-411e-b72d-2010871edf08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e34de96-6bf5-411e-b72d-2010871edf08-istiod-ca-cert\") pod \"3e34de96-6bf5-411e-b72d-2010871edf08\" (UID: \"3e34de96-6bf5-411e-b72d-2010871edf08\") " Nov 01 22:45:23.829051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.828966 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-credential-socket" pod "3e34de96-6bf5-411e-b72d-2010871edf08" (UID: "3e34de96-6bf5-411e-b72d-2010871edf08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:23.829321 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.829299 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e34de96-6bf5-411e-b72d-2010871edf08-istiod-ca-cert" pod "3e34de96-6bf5-411e-b72d-2010871edf08" (UID: "3e34de96-6bf5-411e-b72d-2010871edf08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:23.829741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.829718 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:23.829890 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.829747 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:23.829890 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.829762 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:23.829890 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.829776 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:23.829890 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.829827 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e34de96-6bf5-411e-b72d-2010871edf08-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:23.830787 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.830700 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-istio-token" pod "3e34de96-6bf5-411e-b72d-2010871edf08" (UID: "3e34de96-6bf5-411e-b72d-2010871edf08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:23.831049 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.831024 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e34de96-6bf5-411e-b72d-2010871edf08-istio-podinfo" pod "3e34de96-6bf5-411e-b72d-2010871edf08" (UID: "3e34de96-6bf5-411e-b72d-2010871edf08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:23.831182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.831159 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-kube-api-access-ltjdx" pod "3e34de96-6bf5-411e-b72d-2010871edf08" (UID: "3e34de96-6bf5-411e-b72d-2010871edf08"). InnerVolumeSpecName "kube-api-access-ltjdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:23.831270 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.831255 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-envoy" pod "3e34de96-6bf5-411e-b72d-2010871edf08" (UID: "3e34de96-6bf5-411e-b72d-2010871edf08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:23.932238 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.932141 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e34de96-6bf5-411e-b72d-2010871edf08-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:23.932238 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.932180 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:23.932238 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.932195 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e34de96-6bf5-411e-b72d-2010871edf08-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:23.932238 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:23.932208 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltjdx\" (UniqueName: \"kubernetes.io/projected/3e34de96-6bf5-411e-b72d-2010871edf08-kube-api-access-ltjdx\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.083023 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.082982 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq"] Nov 01 22:45:24.084532 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084502 6466 cpu_manager.go:414] "containerMap: removing container" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" Nov 01 22:45:24.084532 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084534 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" Nov 01 22:45:24.084703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084571 6466 cpu_manager.go:414] "containerMap: removing container" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" Nov 01 22:45:24.084703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084579 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" Nov 01 22:45:24.084703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084591 6466 cpu_manager.go:414] "containerMap: removing container" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" Nov 01 22:45:24.084703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084597 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" Nov 01 22:45:24.084703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084621 6466 cpu_manager.go:414] "containerMap: removing container" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" Nov 01 22:45:24.084703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084629 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" Nov 01 22:45:24.084703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084668 6466 cpu_manager.go:414] "containerMap: removing container" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerName="istio-proxy" Nov 01 22:45:24.084703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084686 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerName="istio-proxy" Nov 01 22:45:24.084956 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084939 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c86719-9d61-4c99-b28f-6eba7ebe6ae2" containerName="istio-proxy" Nov 01 22:45:24.084991 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084964 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" Nov 01 22:45:24.084991 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084977 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" containerName="istio-proxy" Nov 01 22:45:24.085053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.084990 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff42c4d5-6fe0-4a47-b371-238761f656dc" containerName="istio-proxy" Nov 01 22:45:24.085053 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.085004 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a544a3-27d3-43ff-98c4-b47975cfc225" containerName="istio-proxy" Nov 01 22:45:24.091399 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.091373 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.097787 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.097759 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq"] Nov 01 22:45:24.101919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.101895 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1fddbd-serving-cert\"" Nov 01 22:45:24.136870 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.136838 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-metrics-certs\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.137141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.137087 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdfhv\" (UniqueName: \"kubernetes.io/projected/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-kube-api-access-xdfhv\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.137222 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.137189 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.137289 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.137278 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.242178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.241271 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-metrics-certs\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.242178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.241559 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdfhv\" (UniqueName: \"kubernetes.io/projected/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-kube-api-access-xdfhv\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.242178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.241607 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.242178 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.241763 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.261147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.259928 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-maas-api-tls\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.261147 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.260861 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.279208 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.279076 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-metrics-certs\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.304129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.297897 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdfhv\" (UniqueName: \"kubernetes.io/projected/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-kube-api-access-xdfhv\") pod \"maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.418485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.418449 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:24.478671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.478622 6466 generic.go:356] "Generic (PLEG): container finished" podID="43636efd-d17f-4e43-bb3d-f82b63999337" containerID="7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c" exitCode=0 Nov 01 22:45:24.481069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.478791 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" event={"ID":"43636efd-d17f-4e43-bb3d-f82b63999337","Type":"ContainerDied","Data":"7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c"} Nov 01 22:45:24.481069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.479347 6466 scope.go:122] "RemoveContainer" containerID="7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c" Nov 01 22:45:24.484287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.481976 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:45:24.484287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.482089 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:45:24.484287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.482262 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:45:24.484287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.482758 6466 generic.go:356] "Generic (PLEG): container finished" podID="3e34de96-6bf5-411e-b72d-2010871edf08" containerID="b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603" exitCode=0 Nov 01 22:45:24.484287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.482843 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" event={"ID":"3e34de96-6bf5-411e-b72d-2010871edf08","Type":"ContainerDied","Data":"b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603"} Nov 01 22:45:24.484287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.482878 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" event={"ID":"3e34de96-6bf5-411e-b72d-2010871edf08","Type":"ContainerDied","Data":"be0f5bcc9e828af970ae29efdf7c920ca232169d0b9a4f33f2bf53fe7f172861"} Nov 01 22:45:24.484287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.482904 6466 scope.go:122] "RemoveContainer" containerID="b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603" Nov 01 22:45:24.484287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.482984 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g" Nov 01 22:45:24.507871 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.507843 6466 scope.go:122] "RemoveContainer" containerID="f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360" Nov 01 22:45:24.531386 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.531350 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g"] Nov 01 22:45:24.537743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.537711 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b4f6bb-openshift-default-565964c695-trb8g"] Nov 01 22:45:24.549271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.549251 6466 scope.go:122] "RemoveContainer" containerID="b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603" Nov 01 22:45:24.549648 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:24.549623 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603\": container with ID starting with b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603 not found: ID does not exist" containerID="b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603" Nov 01 22:45:24.549756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.549662 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603"} err="failed to get container status \"b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603\": rpc error: code = NotFound desc = could not find container \"b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603\": container with ID starting with b9faad365dc2c5c79d522b40ae3c8a65e62e43cb3fbf3a27b3ddc29b63656603 not found: ID does not exist" Nov 01 22:45:24.549756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.549685 6466 scope.go:122] "RemoveContainer" containerID="f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360" Nov 01 22:45:24.550006 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:24.549976 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360\": container with ID starting with f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360 not found: ID does not exist" containerID="f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360" Nov 01 22:45:24.550152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.550016 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360"} err="failed to get container status \"f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360\": rpc error: code = NotFound desc = could not find container \"f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360\": container with ID starting with f93aafe83dd5d5bfb9ef09030c5b533c08d7b37789ef40f30c0c684ea2aac360 not found: ID does not exist" Nov 01 22:45:24.791304 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:24.790703 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq"] Nov 01 22:45:25.495642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:25.495604 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" event={"ID":"d12d1abf-5fff-4aad-b8c1-04aa3beaa981","Type":"ContainerStarted","Data":"ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff"} Nov 01 22:45:25.496347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:25.496202 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" event={"ID":"d12d1abf-5fff-4aad-b8c1-04aa3beaa981","Type":"ContainerStarted","Data":"de4de15ba6a0ba21d2cd342e18f4ceb203d195375f8bdd21c6f7522e8998e5b2"} Nov 01 22:45:25.496750 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:25.496703 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:25.498534 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:25.498514 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" event={"ID":"43636efd-d17f-4e43-bb3d-f82b63999337","Type":"ContainerStarted","Data":"9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f"} Nov 01 22:45:25.519299 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:25.519188 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" podStartSLOduration=1.060139799 podStartE2EDuration="1.519100171s" totalImagesPullingTime="458.960372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:25.516883226 +0000 UTC m=+1329.301322154" watchObservedRunningTime="2026-11-01 22:45:25.519100171 +0000 UTC m=+1329.303539143" Nov 01 22:45:26.183382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:26.183345 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:45:26.184931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:26.184907 6466 patch_prober.go:29] interesting pod/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:45:26.184995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:26.184958 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 01 22:45:26.367473 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:26.367440 6466 scope.go:122] "RemoveContainer" containerID="37d1a84ed3dd4fcb56315922ab5d9269b698349c4088ad1f459fee594932f168" Nov 01 22:45:26.370664 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:26.369826 6466 scope.go:122] "RemoveContainer" containerID="c06c77feb7d123676b8e158be47c0c770e8f89768feb1919aaf2f4a8ac90d3d6" Nov 01 22:45:26.372724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:26.372705 6466 scope.go:122] "RemoveContainer" containerID="9ddc3b9f300988db7ddd95653fcae57c8ec5f11907c024b734f18c3cd939babb" Nov 01 22:45:26.383189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:26.383164 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e34de96-6bf5-411e-b72d-2010871edf08" path="/var/lib/kubelet/pods/3e34de96-6bf5-411e-b72d-2010871edf08/volumes" Nov 01 22:45:27.185132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:27.184512 6466 patch_prober.go:29] interesting pod/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:45:27.185132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:27.184577 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 01 22:45:27.334278 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:27.334250 6466 scope.go:122] "RemoveContainer" containerID="d70002ea99a4945fd8b35457368ea33410294267f7698ee5a275ba71dadfb4db" Nov 01 22:45:27.518422 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:27.518338 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" event={"ID":"d680f9ed-27fd-44d0-a76c-1979761388a0","Type":"ContainerStarted","Data":"5751c9d08183e6b0579e6b01cc9c26e48849619141792e16dcf39d3f6fc17081"} Nov 01 22:45:27.520714 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:27.520686 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" event={"ID":"5d6d8cdd-7079-4eb3-8701-d26549713a55","Type":"ContainerStarted","Data":"0983410b130335e415d9e2837156569d57d2943d67b31cbb256082ea794900d2"} Nov 01 22:45:27.522804 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:27.522780 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" event={"ID":"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e","Type":"ContainerStarted","Data":"a1792b267b613b23f5b413e6fb6479721b35b05465b5303eea2ad1a7ca51c861"} Nov 01 22:45:27.713625 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:27.713590 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr"] Nov 01 22:45:28.189129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.188130 6466 patch_prober.go:29] interesting pod/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:45:28.189129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.188196 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 01 22:45:28.350142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.349329 6466 scope.go:122] "RemoveContainer" containerID="e7cd3bd64260a4d5f97ec73b8d05bce4b24167b930123a97f174f5eed77d9f5f" Nov 01 22:45:28.369131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.365503 6466 scope.go:122] "RemoveContainer" containerID="06592c7757bec3cdce05c5bb910bc196a35306c417392ef1c99234aeeac757dd" Nov 01 22:45:28.692894 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.692859 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn"] Nov 01 22:45:28.694027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.693998 6466 generic.go:356] "Generic (PLEG): container finished" podID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerID="0983410b130335e415d9e2837156569d57d2943d67b31cbb256082ea794900d2" exitCode=60 Nov 01 22:45:28.694134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.694095 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" event={"ID":"5d6d8cdd-7079-4eb3-8701-d26549713a55","Type":"ContainerDied","Data":"0983410b130335e415d9e2837156569d57d2943d67b31cbb256082ea794900d2"} Nov 01 22:45:28.694188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.694170 6466 scope.go:122] "RemoveContainer" containerID="9ddc3b9f300988db7ddd95653fcae57c8ec5f11907c024b734f18c3cd939babb" Nov 01 22:45:28.694610 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.694570 6466 scope.go:122] "RemoveContainer" containerID="0983410b130335e415d9e2837156569d57d2943d67b31cbb256082ea794900d2" Nov 01 22:45:28.694965 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:28.694943 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56_odh-ai-gateway-infra(5d6d8cdd-7079-4eb3-8701-d26549713a55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" Nov 01 22:45:28.698976 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.698953 6466 generic.go:356] "Generic (PLEG): container finished" podID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerID="a1792b267b613b23f5b413e6fb6479721b35b05465b5303eea2ad1a7ca51c861" exitCode=60 Nov 01 22:45:28.699121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.699088 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" event={"ID":"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e","Type":"ContainerDied","Data":"a1792b267b613b23f5b413e6fb6479721b35b05465b5303eea2ad1a7ca51c861"} Nov 01 22:45:28.699577 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.699562 6466 scope.go:122] "RemoveContainer" containerID="a1792b267b613b23f5b413e6fb6479721b35b05465b5303eea2ad1a7ca51c861" Nov 01 22:45:28.700014 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:28.699996 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp_odh-ai-gateway-infra(1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" Nov 01 22:45:28.702318 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.702295 6466 generic.go:356] "Generic (PLEG): container finished" podID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerID="5751c9d08183e6b0579e6b01cc9c26e48849619141792e16dcf39d3f6fc17081" exitCode=60 Nov 01 22:45:28.702421 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.702379 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" event={"ID":"d680f9ed-27fd-44d0-a76c-1979761388a0","Type":"ContainerDied","Data":"5751c9d08183e6b0579e6b01cc9c26e48849619141792e16dcf39d3f6fc17081"} Nov 01 22:45:28.721042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.721019 6466 scope.go:122] "RemoveContainer" containerID="37d1a84ed3dd4fcb56315922ab5d9269b698349c4088ad1f459fee594932f168" Nov 01 22:45:28.722661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.722637 6466 generic.go:356] "Generic (PLEG): container finished" podID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerID="7d773a23a3713be18a055712fffd59a6d81f4028d04cb701e2b0820d79fc97e3" exitCode=6 Nov 01 22:45:28.722783 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.722764 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" event={"ID":"816a7bec-ff31-49a0-9b0c-0112e73b1a5a","Type":"ContainerDied","Data":"7d773a23a3713be18a055712fffd59a6d81f4028d04cb701e2b0820d79fc97e3"} Nov 01 22:45:28.726819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.726794 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp"] Nov 01 22:45:28.731189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.731158 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56"] Nov 01 22:45:28.775807 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.775783 6466 scope.go:122] "RemoveContainer" containerID="c06c77feb7d123676b8e158be47c0c770e8f89768feb1919aaf2f4a8ac90d3d6" Nov 01 22:45:28.794333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:28.794309 6466 scope.go:122] "RemoveContainer" containerID="d70002ea99a4945fd8b35457368ea33410294267f7698ee5a275ba71dadfb4db" Nov 01 22:45:29.067181 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.049600 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" Nov 01 22:45:29.077893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.077865 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-maas-api-service-ca\") pod \"816a7bec-ff31-49a0-9b0c-0112e73b1a5a\" (UID: \"816a7bec-ff31-49a0-9b0c-0112e73b1a5a\") " Nov 01 22:45:29.078034 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.077990 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-kube-api-access-fssg2\" (UniqueName: \"kubernetes.io/projected/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-kube-api-access-fssg2\") pod \"816a7bec-ff31-49a0-9b0c-0112e73b1a5a\" (UID: \"816a7bec-ff31-49a0-9b0c-0112e73b1a5a\") " Nov 01 22:45:29.078491 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.078428 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-maas-api-service-ca" pod "816a7bec-ff31-49a0-9b0c-0112e73b1a5a" (UID: "816a7bec-ff31-49a0-9b0c-0112e73b1a5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:29.079257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.079130 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-maas-api-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:29.090447 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.090420 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-kube-api-access-fssg2" pod "816a7bec-ff31-49a0-9b0c-0112e73b1a5a" (UID: "816a7bec-ff31-49a0-9b0c-0112e73b1a5a"). InnerVolumeSpecName "kube-api-access-fssg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:29.186716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.185779 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fssg2\" (UniqueName: \"kubernetes.io/projected/816a7bec-ff31-49a0-9b0c-0112e73b1a5a-kube-api-access-fssg2\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:29.186716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.186160 6466 patch_prober.go:29] interesting pod/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:45:29.186716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.186215 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 01 22:45:29.298545 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.298457 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh"] Nov 01 22:45:29.298545 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.298546 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v"] Nov 01 22:45:29.334013 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.333970 6466 scope.go:122] "RemoveContainer" containerID="33497e6901ac89a9904f8a11e574694a2df9fc4fddf7b3f5b93868159acca2d3" Nov 01 22:45:29.392536 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.392504 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" Nov 01 22:45:29.540133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.538158 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d680f9ed-27fd-44d0-a76c-1979761388a0-kube-api-access-j6gwm\" (UniqueName: \"kubernetes.io/projected/d680f9ed-27fd-44d0-a76c-1979761388a0-kube-api-access-j6gwm\") pod \"d680f9ed-27fd-44d0-a76c-1979761388a0\" (UID: \"d680f9ed-27fd-44d0-a76c-1979761388a0\") " Nov 01 22:45:29.540133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.538303 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d680f9ed-27fd-44d0-a76c-1979761388a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d680f9ed-27fd-44d0-a76c-1979761388a0-maas-api-service-ca\") pod \"d680f9ed-27fd-44d0-a76c-1979761388a0\" (UID: \"d680f9ed-27fd-44d0-a76c-1979761388a0\") " Nov 01 22:45:29.664134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.662771 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d680f9ed-27fd-44d0-a76c-1979761388a0-maas-api-service-ca" pod "d680f9ed-27fd-44d0-a76c-1979761388a0" (UID: "d680f9ed-27fd-44d0-a76c-1979761388a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:29.775179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.758293 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d680f9ed-27fd-44d0-a76c-1979761388a0-kube-api-access-j6gwm" pod "d680f9ed-27fd-44d0-a76c-1979761388a0" (UID: "d680f9ed-27fd-44d0-a76c-1979761388a0"). InnerVolumeSpecName "kube-api-access-j6gwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:29.775179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.759907 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6gwm\" (UniqueName: \"kubernetes.io/projected/d680f9ed-27fd-44d0-a76c-1979761388a0-kube-api-access-j6gwm\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:29.775179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.759939 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d680f9ed-27fd-44d0-a76c-1979761388a0-maas-api-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:29.903753 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.902854 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" event={"ID":"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7","Type":"ContainerStarted","Data":"a88ddab66acc9df71edc4f5534a470916e858f08ff31167d6e7eda78581e8c26"} Nov 01 22:45:29.912803 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.912772 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" event={"ID":"db9d81f5-1e85-41b2-9eac-c43ccfb98773","Type":"ContainerStarted","Data":"10c2da66255d237a67b8bb406d8ceef53cbfba8528a2f0a3814ff75b18972d14"} Nov 01 22:45:29.914539 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.914483 6466 scope.go:122] "RemoveContainer" containerID="10c2da66255d237a67b8bb406d8ceef53cbfba8528a2f0a3814ff75b18972d14" Nov 01 22:45:29.915042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.915014 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" event={"ID":"d680f9ed-27fd-44d0-a76c-1979761388a0","Type":"ContainerDied","Data":"9e488d9c390b604f2be6595c6a0d8d665cdd4fef5d6e406e0f2fe21278aed792"} Nov 01 22:45:29.915158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.915070 6466 scope.go:122] "RemoveContainer" containerID="5751c9d08183e6b0579e6b01cc9c26e48849619141792e16dcf39d3f6fc17081" Nov 01 22:45:29.915228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.915215 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn" Nov 01 22:45:29.915327 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:29.915293 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf_odh-ai-gateway-infra(db9d81f5-1e85-41b2-9eac-c43ccfb98773)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" Nov 01 22:45:29.919931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.919745 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" event={"ID":"816a7bec-ff31-49a0-9b0c-0112e73b1a5a","Type":"ContainerDied","Data":"1e14328d65a854aa06edcffae6b712d285b374b8514fe70466eac75343d2eac4"} Nov 01 22:45:29.919931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.919817 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr" Nov 01 22:45:29.981159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.978466 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:45:29.982738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:29.982718 6466 scope.go:122] "RemoveContainer" containerID="7d773a23a3713be18a055712fffd59a6d81f4028d04cb701e2b0820d79fc97e3" Nov 01 22:45:30.188797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.187038 6466 patch_prober.go:29] interesting pod/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:45:30.188797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.187097 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 01 22:45:30.406966 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.406939 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn"] Nov 01 22:45:30.407391 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.407375 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93d36a-29892885-s5nbn"] Nov 01 22:45:30.408664 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.408640 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr"] Nov 01 22:45:30.413236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.413217 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" Nov 01 22:45:30.413325 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.413288 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-3e228e-29892885-fqdfr"] Nov 01 22:45:30.500244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.500216 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" Nov 01 22:45:30.594803 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.594772 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-kube-api-access-nwb6b\" (UniqueName: \"kubernetes.io/projected/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-kube-api-access-nwb6b\") pod \"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e\" (UID: \"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e\") " Nov 01 22:45:30.595075 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.595062 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-maas-api-service-ca\") pod \"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e\" (UID: \"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e\") " Nov 01 22:45:30.595146 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.595094 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6d8cdd-7079-4eb3-8701-d26549713a55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6d8cdd-7079-4eb3-8701-d26549713a55-maas-api-service-ca\") pod \"5d6d8cdd-7079-4eb3-8701-d26549713a55\" (UID: \"5d6d8cdd-7079-4eb3-8701-d26549713a55\") " Nov 01 22:45:30.595205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.595153 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6d8cdd-7079-4eb3-8701-d26549713a55-kube-api-access-v9vgt\" (UniqueName: \"kubernetes.io/projected/5d6d8cdd-7079-4eb3-8701-d26549713a55-kube-api-access-v9vgt\") pod \"5d6d8cdd-7079-4eb3-8701-d26549713a55\" (UID: \"5d6d8cdd-7079-4eb3-8701-d26549713a55\") " Nov 01 22:45:30.595406 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.595375 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-maas-api-service-ca" pod "1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" (UID: "1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:30.595520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.595500 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6d8cdd-7079-4eb3-8701-d26549713a55-maas-api-service-ca" pod "5d6d8cdd-7079-4eb3-8701-d26549713a55" (UID: "5d6d8cdd-7079-4eb3-8701-d26549713a55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:30.595887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.595860 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-maas-api-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:30.595887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.595884 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6d8cdd-7079-4eb3-8701-d26549713a55-maas-api-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:30.597093 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.597074 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-kube-api-access-nwb6b" pod "1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" (UID: "1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e"). InnerVolumeSpecName "kube-api-access-nwb6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:30.597345 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.597326 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6d8cdd-7079-4eb3-8701-d26549713a55-kube-api-access-v9vgt" pod "5d6d8cdd-7079-4eb3-8701-d26549713a55" (UID: "5d6d8cdd-7079-4eb3-8701-d26549713a55"). InnerVolumeSpecName "kube-api-access-v9vgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:30.697738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.697700 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9vgt\" (UniqueName: \"kubernetes.io/projected/5d6d8cdd-7079-4eb3-8701-d26549713a55-kube-api-access-v9vgt\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:30.697738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:30.697731 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwb6b\" (UniqueName: \"kubernetes.io/projected/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e-kube-api-access-nwb6b\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:31.017463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.017350 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf"] Nov 01 22:45:31.050144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.033017 6466 generic.go:356] "Generic (PLEG): container finished" podID="14785ed4-8e76-4509-9712-0026d2e80b74" containerID="609d8e9865737e32bd34cbde8350ba1f50875c17ae5b02a59aab76da988f9d70" exitCode=60 Nov 01 22:45:31.050144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.033196 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" event={"ID":"14785ed4-8e76-4509-9712-0026d2e80b74","Type":"ContainerDied","Data":"609d8e9865737e32bd34cbde8350ba1f50875c17ae5b02a59aab76da988f9d70"} Nov 01 22:45:31.050144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.033258 6466 scope.go:122] "RemoveContainer" containerID="33497e6901ac89a9904f8a11e574694a2df9fc4fddf7b3f5b93868159acca2d3" Nov 01 22:45:31.050144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.033824 6466 scope.go:122] "RemoveContainer" containerID="609d8e9865737e32bd34cbde8350ba1f50875c17ae5b02a59aab76da988f9d70" Nov 01 22:45:31.050144 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:31.034317 6466 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j_odh-ai-gateway-infra(14785ed4-8e76-4509-9712-0026d2e80b74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" Nov 01 22:45:31.109033 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.104296 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" Nov 01 22:45:31.109033 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.105389 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56" event={"ID":"5d6d8cdd-7079-4eb3-8701-d26549713a55","Type":"ContainerDied","Data":"7e5f7854914a41a258ea95ac3a7a49e09cacdc6596bc304c2c1d168fc6228a7d"} Nov 01 22:45:31.111768 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.110708 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf"] Nov 01 22:45:31.113746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.113561 6466 generic.go:356] "Generic (PLEG): container finished" podID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerID="a88ddab66acc9df71edc4f5534a470916e858f08ff31167d6e7eda78581e8c26" exitCode=60 Nov 01 22:45:31.113746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.113652 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" event={"ID":"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7","Type":"ContainerDied","Data":"a88ddab66acc9df71edc4f5534a470916e858f08ff31167d6e7eda78581e8c26"} Nov 01 22:45:31.116202 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.116179 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" Nov 01 22:45:31.116202 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.116194 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp" event={"ID":"1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e","Type":"ContainerDied","Data":"8e3480922386d4972d05e24bd9724ce3afff2d3258449e135fc5efede2865032"} Nov 01 22:45:31.121289 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.120635 6466 generic.go:356] "Generic (PLEG): container finished" podID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerID="10c2da66255d237a67b8bb406d8ceef53cbfba8528a2f0a3814ff75b18972d14" exitCode=60 Nov 01 22:45:31.121289 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.120864 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" event={"ID":"db9d81f5-1e85-41b2-9eac-c43ccfb98773","Type":"ContainerDied","Data":"10c2da66255d237a67b8bb406d8ceef53cbfba8528a2f0a3814ff75b18972d14"} Nov 01 22:45:31.125995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.125966 6466 scope.go:122] "RemoveContainer" containerID="0983410b130335e415d9e2837156569d57d2943d67b31cbb256082ea794900d2" Nov 01 22:45:31.184144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.183748 6466 patch_prober.go:29] interesting pod/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:45:31.184144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.183803 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 01 22:45:31.186705 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.186677 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp"] Nov 01 22:45:31.187852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.187748 6466 scope.go:122] "RemoveContainer" containerID="e7cd3bd64260a4d5f97ec73b8d05bce4b24167b930123a97f174f5eed77d9f5f" Nov 01 22:45:31.193497 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.193472 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0fd159-29892885-zxptp"] Nov 01 22:45:31.214312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.212806 6466 scope.go:122] "RemoveContainer" containerID="a1792b267b613b23f5b413e6fb6479721b35b05465b5303eea2ad1a7ca51c861" Nov 01 22:45:31.223021 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.222994 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56"] Nov 01 22:45:31.223235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.223222 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-b28a40-29892885-kbz56"] Nov 01 22:45:31.286486 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.286164 6466 scope.go:122] "RemoveContainer" containerID="06592c7757bec3cdce05c5bb910bc196a35306c417392ef1c99234aeeac757dd" Nov 01 22:45:31.428559 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.428530 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" Nov 01 22:45:31.475945 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.475912 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" Nov 01 22:45:31.517875 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.517844 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:45:31.522939 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.522913 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-maas-api-service-ca\") pod \"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7\" (UID: \"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7\") " Nov 01 22:45:31.523251 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.523231 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-kube-api-access-qqqkh\" (UniqueName: \"kubernetes.io/projected/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-kube-api-access-qqqkh\") pod \"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7\" (UID: \"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7\") " Nov 01 22:45:31.523348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.523312 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-maas-api-service-ca" pod "7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" (UID: "7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:31.523781 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.523761 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-maas-api-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:31.525444 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.525421 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-kube-api-access-qqqkh" pod "7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" (UID: "7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7"). InnerVolumeSpecName "kube-api-access-qqqkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:31.624663 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.624568 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9d81f5-1e85-41b2-9eac-c43ccfb98773-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9d81f5-1e85-41b2-9eac-c43ccfb98773-maas-api-service-ca\") pod \"db9d81f5-1e85-41b2-9eac-c43ccfb98773\" (UID: \"db9d81f5-1e85-41b2-9eac-c43ccfb98773\") " Nov 01 22:45:31.624663 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.624652 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9d81f5-1e85-41b2-9eac-c43ccfb98773-kube-api-access-vtqj6\" (UniqueName: \"kubernetes.io/projected/db9d81f5-1e85-41b2-9eac-c43ccfb98773-kube-api-access-vtqj6\") pod \"db9d81f5-1e85-41b2-9eac-c43ccfb98773\" (UID: \"db9d81f5-1e85-41b2-9eac-c43ccfb98773\") " Nov 01 22:45:31.624977 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.624952 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9d81f5-1e85-41b2-9eac-c43ccfb98773-maas-api-service-ca" pod "db9d81f5-1e85-41b2-9eac-c43ccfb98773" (UID: "db9d81f5-1e85-41b2-9eac-c43ccfb98773"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:31.625908 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.625881 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqqkh\" (UniqueName: \"kubernetes.io/projected/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7-kube-api-access-qqqkh\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:31.626040 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.625912 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9d81f5-1e85-41b2-9eac-c43ccfb98773-maas-api-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:31.627801 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.627777 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9d81f5-1e85-41b2-9eac-c43ccfb98773-kube-api-access-vtqj6" pod "db9d81f5-1e85-41b2-9eac-c43ccfb98773" (UID: "db9d81f5-1e85-41b2-9eac-c43ccfb98773"). InnerVolumeSpecName "kube-api-access-vtqj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:31.728444 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:31.728399 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtqj6\" (UniqueName: \"kubernetes.io/projected/db9d81f5-1e85-41b2-9eac-c43ccfb98773-kube-api-access-vtqj6\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:32.094134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.089845 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j"] Nov 01 22:45:32.142959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.142933 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" Nov 01 22:45:32.142959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.142938 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf" event={"ID":"7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7","Type":"ContainerDied","Data":"ad73ef8c046152a70fc025226363295adc662941a67a891b40c0e7db077f4e29"} Nov 01 22:45:32.143201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.143007 6466 scope.go:122] "RemoveContainer" containerID="a88ddab66acc9df71edc4f5534a470916e858f08ff31167d6e7eda78581e8c26" Nov 01 22:45:32.152497 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.152466 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" event={"ID":"db9d81f5-1e85-41b2-9eac-c43ccfb98773","Type":"ContainerDied","Data":"0135c7544d62f3f67cb3cd89e30d3fc6df6ca4f5aba08f0d0b2d288f0c8447ff"} Nov 01 22:45:32.152650 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.152636 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf" Nov 01 22:45:32.168995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.168974 6466 scope.go:122] "RemoveContainer" containerID="10c2da66255d237a67b8bb406d8ceef53cbfba8528a2f0a3814ff75b18972d14" Nov 01 22:45:32.188353 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.188325 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:45:32.189483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.189464 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:45:32.195797 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.195775 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf"] Nov 01 22:45:32.205995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.205968 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a9c08e-29892885-5nqxf"] Nov 01 22:45:32.230204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.230176 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf"] Nov 01 22:45:32.236662 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.236587 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3fa400-29892885-f9tcf"] Nov 01 22:45:32.359378 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.351416 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" path="/var/lib/kubelet/pods/1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e/volumes" Nov 01 22:45:32.359378 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.352131 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" path="/var/lib/kubelet/pods/5d6d8cdd-7079-4eb3-8701-d26549713a55/volumes" Nov 01 22:45:32.359378 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.352732 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" path="/var/lib/kubelet/pods/7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7/volumes" Nov 01 22:45:32.359378 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.353727 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" path="/var/lib/kubelet/pods/816a7bec-ff31-49a0-9b0c-0112e73b1a5a/volumes" Nov 01 22:45:32.359378 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.354312 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" path="/var/lib/kubelet/pods/d680f9ed-27fd-44d0-a76c-1979761388a0/volumes" Nov 01 22:45:32.359378 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.354854 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" path="/var/lib/kubelet/pods/db9d81f5-1e85-41b2-9eac-c43ccfb98773/volumes" Nov 01 22:45:32.382483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.382454 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" Nov 01 22:45:32.543607 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.543577 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14785ed4-8e76-4509-9712-0026d2e80b74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14785ed4-8e76-4509-9712-0026d2e80b74-maas-api-service-ca\") pod \"14785ed4-8e76-4509-9712-0026d2e80b74\" (UID: \"14785ed4-8e76-4509-9712-0026d2e80b74\") " Nov 01 22:45:32.543969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.543710 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14785ed4-8e76-4509-9712-0026d2e80b74-kube-api-access-pnfs2\" (UniqueName: \"kubernetes.io/projected/14785ed4-8e76-4509-9712-0026d2e80b74-kube-api-access-pnfs2\") pod \"14785ed4-8e76-4509-9712-0026d2e80b74\" (UID: \"14785ed4-8e76-4509-9712-0026d2e80b74\") " Nov 01 22:45:32.543969 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.543951 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14785ed4-8e76-4509-9712-0026d2e80b74-maas-api-service-ca" pod "14785ed4-8e76-4509-9712-0026d2e80b74" (UID: "14785ed4-8e76-4509-9712-0026d2e80b74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:32.544474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.544457 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14785ed4-8e76-4509-9712-0026d2e80b74-maas-api-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:32.545599 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.545580 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14785ed4-8e76-4509-9712-0026d2e80b74-kube-api-access-pnfs2" pod "14785ed4-8e76-4509-9712-0026d2e80b74" (UID: "14785ed4-8e76-4509-9712-0026d2e80b74"). InnerVolumeSpecName "kube-api-access-pnfs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:32.646600 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:32.646552 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnfs2\" (UniqueName: \"kubernetes.io/projected/14785ed4-8e76-4509-9712-0026d2e80b74-kube-api-access-pnfs2\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:33.165000 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:33.164971 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" Nov 01 22:45:33.165000 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:33.164988 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j" event={"ID":"14785ed4-8e76-4509-9712-0026d2e80b74","Type":"ContainerDied","Data":"6c71954b0487b9e5c3df56f658a5a7e1c3f38c27feedb6f7e200d3632fe44f16"} Nov 01 22:45:33.165282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:33.165055 6466 scope.go:122] "RemoveContainer" containerID="609d8e9865737e32bd34cbde8350ba1f50875c17ae5b02a59aab76da988f9d70" Nov 01 22:45:33.197676 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:33.197639 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j"] Nov 01 22:45:33.203007 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:33.202979 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-aa441b-29892885-hzx2j"] Nov 01 22:45:34.299134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.299071 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" podUID="a1e35591-1c5a-485f-aea5-064e48802a9b" containerName="payload-processing" containerID="cri-o://31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a" gracePeriod=25 Nov 01 22:45:34.299543 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.299083 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" podUID="42b08c73-7108-40f6-b0e0-456446484085" containerName="payload-pre-processing" containerID="cri-o://b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a" gracePeriod=25 Nov 01 22:45:34.343737 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.343710 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" path="/var/lib/kubelet/pods/14785ed4-8e76-4509-9712-0026d2e80b74/volumes" Nov 01 22:45:34.603670 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.603639 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:45:34.626540 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.626509 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:45:34.806714 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.806680 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b08c73-7108-40f6-b0e0-456446484085-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b08c73-7108-40f6-b0e0-456446484085-plugins-config-volume\") pod \"42b08c73-7108-40f6-b0e0-456446484085\" (UID: \"42b08c73-7108-40f6-b0e0-456446484085\") " Nov 01 22:45:34.806908 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.806728 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e35591-1c5a-485f-aea5-064e48802a9b-kube-api-access-vcwcs\" (UniqueName: \"kubernetes.io/projected/a1e35591-1c5a-485f-aea5-064e48802a9b-kube-api-access-vcwcs\") pod \"a1e35591-1c5a-485f-aea5-064e48802a9b\" (UID: \"a1e35591-1c5a-485f-aea5-064e48802a9b\") " Nov 01 22:45:34.806908 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.806779 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b08c73-7108-40f6-b0e0-456446484085-kube-api-access-pxxgj\" (UniqueName: \"kubernetes.io/projected/42b08c73-7108-40f6-b0e0-456446484085-kube-api-access-pxxgj\") pod \"42b08c73-7108-40f6-b0e0-456446484085\" (UID: \"42b08c73-7108-40f6-b0e0-456446484085\") " Nov 01 22:45:34.807016 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.807000 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e35591-1c5a-485f-aea5-064e48802a9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e35591-1c5a-485f-aea5-064e48802a9b-plugins-config-volume\") pod \"a1e35591-1c5a-485f-aea5-064e48802a9b\" (UID: \"a1e35591-1c5a-485f-aea5-064e48802a9b\") " Nov 01 22:45:34.807059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.807038 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b08c73-7108-40f6-b0e0-456446484085-plugins-config-volume" pod "42b08c73-7108-40f6-b0e0-456446484085" (UID: "42b08c73-7108-40f6-b0e0-456446484085"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:34.807475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.807447 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e35591-1c5a-485f-aea5-064e48802a9b-plugins-config-volume" pod "a1e35591-1c5a-485f-aea5-064e48802a9b" (UID: "a1e35591-1c5a-485f-aea5-064e48802a9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:34.807816 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.807796 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e35591-1c5a-485f-aea5-064e48802a9b-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:34.807816 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.807817 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b08c73-7108-40f6-b0e0-456446484085-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:34.809051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.809029 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b08c73-7108-40f6-b0e0-456446484085-kube-api-access-pxxgj" pod "42b08c73-7108-40f6-b0e0-456446484085" (UID: "42b08c73-7108-40f6-b0e0-456446484085"). InnerVolumeSpecName "kube-api-access-pxxgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:34.809186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.809067 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e35591-1c5a-485f-aea5-064e48802a9b-kube-api-access-vcwcs" pod "a1e35591-1c5a-485f-aea5-064e48802a9b" (UID: "a1e35591-1c5a-485f-aea5-064e48802a9b"). InnerVolumeSpecName "kube-api-access-vcwcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:34.909455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.909417 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcwcs\" (UniqueName: \"kubernetes.io/projected/a1e35591-1c5a-485f-aea5-064e48802a9b-kube-api-access-vcwcs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:34.909455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:34.909450 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxxgj\" (UniqueName: \"kubernetes.io/projected/42b08c73-7108-40f6-b0e0-456446484085-kube-api-access-pxxgj\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:35.183867 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.183779 6466 generic.go:356] "Generic (PLEG): container finished" podID="42b08c73-7108-40f6-b0e0-456446484085" containerID="b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a" exitCode=0 Nov 01 22:45:35.183867 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.183825 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" Nov 01 22:45:35.184103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.183871 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" event={"ID":"42b08c73-7108-40f6-b0e0-456446484085","Type":"ContainerDied","Data":"b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a"} Nov 01 22:45:35.184103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.183937 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v" event={"ID":"42b08c73-7108-40f6-b0e0-456446484085","Type":"ContainerDied","Data":"750ff3d3bf1f72a0abde505f620ce59f53d84ef09d6f02f933d6050fa5343c61"} Nov 01 22:45:35.184103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.183966 6466 scope.go:122] "RemoveContainer" containerID="b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a" Nov 01 22:45:35.185463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.185440 6466 generic.go:356] "Generic (PLEG): container finished" podID="a1e35591-1c5a-485f-aea5-064e48802a9b" containerID="31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a" exitCode=0 Nov 01 22:45:35.185599 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.185470 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" event={"ID":"a1e35591-1c5a-485f-aea5-064e48802a9b","Type":"ContainerDied","Data":"31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a"} Nov 01 22:45:35.185599 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.185491 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" Nov 01 22:45:35.185599 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.185522 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh" event={"ID":"a1e35591-1c5a-485f-aea5-064e48802a9b","Type":"ContainerDied","Data":"b36e389c1cb4b9d97f62991311b3d120f508a8bea40e800170024045a69250f3"} Nov 01 22:45:35.199275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.199252 6466 scope.go:122] "RemoveContainer" containerID="b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a" Nov 01 22:45:35.201626 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:35.201577 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a\": container with ID starting with b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a not found: ID does not exist" containerID="b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a" Nov 01 22:45:35.201754 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.201627 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a"} err="failed to get container status \"b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a\": rpc error: code = NotFound desc = could not find container \"b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a\": container with ID starting with b2edcde2c27c5c633b5b957425f9c0a3c093d08a995a549694c9243f2c08b79a not found: ID does not exist" Nov 01 22:45:35.201754 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.201651 6466 scope.go:122] "RemoveContainer" containerID="31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a" Nov 01 22:45:35.221121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.221046 6466 scope.go:122] "RemoveContainer" containerID="31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a" Nov 01 22:45:35.221460 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:35.221430 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a\": container with ID starting with 31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a not found: ID does not exist" containerID="31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a" Nov 01 22:45:35.221572 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.221469 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a"} err="failed to get container status \"31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a\": rpc error: code = NotFound desc = could not find container \"31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a\": container with ID starting with 31aea9d23b872019515a3beb19e39249a3195c1b31ea6e74464d3e3b25c89c0a not found: ID does not exist" Nov 01 22:45:35.225123 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.225090 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v"] Nov 01 22:45:35.230889 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.230865 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-10a1d9-c4894546c-75z2v"] Nov 01 22:45:35.239735 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.239427 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh"] Nov 01 22:45:35.243442 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:35.243420 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-10a1d9-6c7cc4c667-p2nhh"] Nov 01 22:45:36.342530 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:36.342488 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b08c73-7108-40f6-b0e0-456446484085" path="/var/lib/kubelet/pods/42b08c73-7108-40f6-b0e0-456446484085/volumes" Nov 01 22:45:36.343300 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:36.343277 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e35591-1c5a-485f-aea5-064e48802a9b" path="/var/lib/kubelet/pods/a1e35591-1c5a-485f-aea5-064e48802a9b/volumes" Nov 01 22:45:36.725096 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:36.725056 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75"] Nov 01 22:45:36.725376 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:36.725345 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" containerID="cri-o://dc9af7de362fa4ad2b33abcd6ba4a74188bdd09692999cb35b550ffb60e97c1d" gracePeriod=30 Nov 01 22:45:39.335090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:39.335052 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9"] Nov 01 22:45:39.335652 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:39.335596 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" containerID="cri-o://739cbf8f725203790b4529424cebe4cfda23a0a58c723ae12ec61a816f04724b" gracePeriod=30 Nov 01 22:45:39.957212 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:39.957176 6466 patch_prober.go:29] interesting pod/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:45:39.957371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:39.957241 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:45:42.458132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.454656 6466 generic.go:356] "Generic (PLEG): container finished" podID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerID="dc9af7de362fa4ad2b33abcd6ba4a74188bdd09692999cb35b550ffb60e97c1d" exitCode=0 Nov 01 22:45:42.716954 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.715482 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:45:42.759849 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.750731 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-kube-api-access-mjm7r\" (UniqueName: \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-kube-api-access-mjm7r\") pod \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " Nov 01 22:45:42.774456 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.773256 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" event={"ID":"2f8221bb-bcff-4e2a-af5c-76f36b0a6477","Type":"ContainerDied","Data":"dc9af7de362fa4ad2b33abcd6ba4a74188bdd09692999cb35b550ffb60e97c1d"} Nov 01 22:45:42.774456 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.773345 6466 scope.go:122] "RemoveContainer" containerID="dc9af7de362fa4ad2b33abcd6ba4a74188bdd09692999cb35b550ffb60e97c1d" Nov 01 22:45:42.776929 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.776325 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-kube-api-access-mjm7r" pod "2f8221bb-bcff-4e2a-af5c-76f36b0a6477" (UID: "2f8221bb-bcff-4e2a-af5c-76f36b0a6477"). InnerVolumeSpecName "kube-api-access-mjm7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:42.798762 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.798742 6466 scope.go:122] "RemoveContainer" containerID="a661736aecfbf2f3003ca2e286777d9b4362dfe43e99d7ef2dcad773b161b487" Nov 01 22:45:42.870363 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.870328 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-certs\") pod \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " Nov 01 22:45:42.870535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.870472 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-token\" (UniqueName: \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-token\") pod \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " Nov 01 22:45:42.870535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.870510 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-podinfo\") pod \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " Nov 01 22:45:42.870637 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.870610 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-envoy\") pod \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " Nov 01 22:45:42.870697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.870666 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-credential-socket\") pod \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " Nov 01 22:45:42.870697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.870664 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-certs" pod "2f8221bb-bcff-4e2a-af5c-76f36b0a6477" (UID: "2f8221bb-bcff-4e2a-af5c-76f36b0a6477"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:42.870811 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.870729 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istiod-ca-cert\") pod \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " Nov 01 22:45:42.870873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.870859 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-socket\") pod \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " Nov 01 22:45:42.870927 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.870888 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-credential-socket" pod "2f8221bb-bcff-4e2a-af5c-76f36b0a6477" (UID: "2f8221bb-bcff-4e2a-af5c-76f36b0a6477"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:42.870995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.870975 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-data\") pod \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\" (UID: \"2f8221bb-bcff-4e2a-af5c-76f36b0a6477\") " Nov 01 22:45:42.871102 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.871082 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-socket" pod "2f8221bb-bcff-4e2a-af5c-76f36b0a6477" (UID: "2f8221bb-bcff-4e2a-af5c-76f36b0a6477"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:42.871345 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.871315 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-data" pod "2f8221bb-bcff-4e2a-af5c-76f36b0a6477" (UID: "2f8221bb-bcff-4e2a-af5c-76f36b0a6477"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:42.871426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.871333 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istiod-ca-cert" pod "2f8221bb-bcff-4e2a-af5c-76f36b0a6477" (UID: "2f8221bb-bcff-4e2a-af5c-76f36b0a6477"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:42.871892 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.871865 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjm7r\" (UniqueName: \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-kube-api-access-mjm7r\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:42.871988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.871910 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:42.871988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.871927 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:42.871988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.871944 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:42.871988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.871958 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:42.871988 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.871972 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:42.873068 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.873042 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-token" pod "2f8221bb-bcff-4e2a-af5c-76f36b0a6477" (UID: "2f8221bb-bcff-4e2a-af5c-76f36b0a6477"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:42.873303 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.873281 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-envoy" pod "2f8221bb-bcff-4e2a-af5c-76f36b0a6477" (UID: "2f8221bb-bcff-4e2a-af5c-76f36b0a6477"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:42.873384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.873284 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-podinfo" pod "2f8221bb-bcff-4e2a-af5c-76f36b0a6477" (UID: "2f8221bb-bcff-4e2a-af5c-76f36b0a6477"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:42.973362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.973326 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:42.973362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.973359 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:42.973567 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:42.973372 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8221bb-bcff-4e2a-af5c-76f36b0a6477-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:43.463913 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:43.463882 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" Nov 01 22:45:43.464297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:43.463887 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75" event={"ID":"2f8221bb-bcff-4e2a-af5c-76f36b0a6477","Type":"ContainerDied","Data":"c38bc7c86f4987cc633b4c146d4c63dacb18a0a327a9aeed898e7afea4fd9e90"} Nov 01 22:45:43.492435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:43.492405 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75"] Nov 01 22:45:43.501852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:43.501824 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3e228e-openshift-default-5675fb8795-52d75"] Nov 01 22:45:44.343413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.343379 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" path="/var/lib/kubelet/pods/2f8221bb-bcff-4e2a-af5c-76f36b0a6477/volumes" Nov 01 22:45:44.473802 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.473775 6466 generic.go:356] "Generic (PLEG): container finished" podID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerID="739cbf8f725203790b4529424cebe4cfda23a0a58c723ae12ec61a816f04724b" exitCode=0 Nov 01 22:45:44.474253 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.473845 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" event={"ID":"1f1280b1-f9ea-489c-8b06-f595fae26903","Type":"ContainerDied","Data":"739cbf8f725203790b4529424cebe4cfda23a0a58c723ae12ec61a816f04724b"} Nov 01 22:45:44.474253 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.473925 6466 scope.go:122] "RemoveContainer" containerID="a0e350bd90b75c131e38853e602f0a1f838db8a93983274d3106b3544cf9c217" Nov 01 22:45:44.625182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.625158 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:45:44.705125 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705081 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-podinfo\") pod \"1f1280b1-f9ea-489c-8b06-f595fae26903\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " Nov 01 22:45:44.705314 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705216 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1280b1-f9ea-489c-8b06-f595fae26903-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1280b1-f9ea-489c-8b06-f595fae26903-istiod-ca-cert\") pod \"1f1280b1-f9ea-489c-8b06-f595fae26903\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " Nov 01 22:45:44.705433 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705419 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-envoy\") pod \"1f1280b1-f9ea-489c-8b06-f595fae26903\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " Nov 01 22:45:44.705490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705469 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-certs\") pod \"1f1280b1-f9ea-489c-8b06-f595fae26903\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " Nov 01 22:45:44.705550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705526 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-socket\") pod \"1f1280b1-f9ea-489c-8b06-f595fae26903\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " Nov 01 22:45:44.705611 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705596 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-token\" (UniqueName: \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-token\") pod \"1f1280b1-f9ea-489c-8b06-f595fae26903\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " Nov 01 22:45:44.705664 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705634 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-data\") pod \"1f1280b1-f9ea-489c-8b06-f595fae26903\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " Nov 01 22:45:44.706090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705695 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-certs" pod "1f1280b1-f9ea-489c-8b06-f595fae26903" (UID: "1f1280b1-f9ea-489c-8b06-f595fae26903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:44.706090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705712 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-kube-api-access-t722x\" (UniqueName: \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-kube-api-access-t722x\") pod \"1f1280b1-f9ea-489c-8b06-f595fae26903\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " Nov 01 22:45:44.706090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705656 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1280b1-f9ea-489c-8b06-f595fae26903-istiod-ca-cert" pod "1f1280b1-f9ea-489c-8b06-f595fae26903" (UID: "1f1280b1-f9ea-489c-8b06-f595fae26903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:44.706090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705745 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-credential-socket\") pod \"1f1280b1-f9ea-489c-8b06-f595fae26903\" (UID: \"1f1280b1-f9ea-489c-8b06-f595fae26903\") " Nov 01 22:45:44.706090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.705931 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-credential-socket" pod "1f1280b1-f9ea-489c-8b06-f595fae26903" (UID: "1f1280b1-f9ea-489c-8b06-f595fae26903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:44.706410 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.706103 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-data" pod "1f1280b1-f9ea-489c-8b06-f595fae26903" (UID: "1f1280b1-f9ea-489c-8b06-f595fae26903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:44.706410 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.706302 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-socket" pod "1f1280b1-f9ea-489c-8b06-f595fae26903" (UID: "1f1280b1-f9ea-489c-8b06-f595fae26903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:44.706846 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.706821 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.706955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.706850 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.706955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.706864 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.706955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.706877 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.706955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.706891 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1280b1-f9ea-489c-8b06-f595fae26903-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.708058 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.708031 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-podinfo" pod "1f1280b1-f9ea-489c-8b06-f595fae26903" (UID: "1f1280b1-f9ea-489c-8b06-f595fae26903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:44.708692 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.708648 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-token" pod "1f1280b1-f9ea-489c-8b06-f595fae26903" (UID: "1f1280b1-f9ea-489c-8b06-f595fae26903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:44.708782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.708758 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-envoy" pod "1f1280b1-f9ea-489c-8b06-f595fae26903" (UID: "1f1280b1-f9ea-489c-8b06-f595fae26903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:44.710221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.710196 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-kube-api-access-t722x" pod "1f1280b1-f9ea-489c-8b06-f595fae26903" (UID: "1f1280b1-f9ea-489c-8b06-f595fae26903"). InnerVolumeSpecName "kube-api-access-t722x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:44.808711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.808669 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.808711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.808707 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t722x\" (UniqueName: \"kubernetes.io/projected/1f1280b1-f9ea-489c-8b06-f595fae26903-kube-api-access-t722x\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.808711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.808718 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.808939 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.808728 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1280b1-f9ea-489c-8b06-f595fae26903-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.935975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.935943 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt"] Nov 01 22:45:44.936790 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936772 6466 cpu_manager.go:414] "containerMap: removing container" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerName="cleanup" Nov 01 22:45:44.936838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936791 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerName="cleanup" Nov 01 22:45:44.936838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936803 6466 cpu_manager.go:414] "containerMap: removing container" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" containerName="cleanup" Nov 01 22:45:44.936838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936808 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" containerName="cleanup" Nov 01 22:45:44.936838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936815 6466 cpu_manager.go:414] "containerMap: removing container" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerName="cleanup" Nov 01 22:45:44.936838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936821 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerName="cleanup" Nov 01 22:45:44.936838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936829 6466 cpu_manager.go:414] "containerMap: removing container" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerName="cleanup" Nov 01 22:45:44.936838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936834 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936846 6466 cpu_manager.go:414] "containerMap: removing container" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936852 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936858 6466 cpu_manager.go:414] "containerMap: removing container" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936862 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936871 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936875 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936883 6466 cpu_manager.go:414] "containerMap: removing container" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936887 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936897 6466 cpu_manager.go:414] "containerMap: removing container" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936902 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936912 6466 cpu_manager.go:414] "containerMap: removing container" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936916 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936923 6466 cpu_manager.go:414] "containerMap: removing container" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936927 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936939 6466 cpu_manager.go:414] "containerMap: removing container" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936943 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936952 6466 cpu_manager.go:414] "containerMap: removing container" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936956 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936964 6466 cpu_manager.go:414] "containerMap: removing container" podUID="42b08c73-7108-40f6-b0e0-456446484085" containerName="payload-pre-processing" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936969 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b08c73-7108-40f6-b0e0-456446484085" containerName="payload-pre-processing" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936975 6466 cpu_manager.go:414] "containerMap: removing container" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936979 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936988 6466 cpu_manager.go:414] "containerMap: removing container" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936992 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.936998 6466 cpu_manager.go:414] "containerMap: removing container" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937002 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937010 6466 cpu_manager.go:414] "containerMap: removing container" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937015 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937021 6466 cpu_manager.go:414] "containerMap: removing container" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937025 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937033 6466 cpu_manager.go:414] "containerMap: removing container" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937036 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937046 6466 cpu_manager.go:414] "containerMap: removing container" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937050 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937058 6466 cpu_manager.go:414] "containerMap: removing container" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937062 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937070 6466 cpu_manager.go:414] "containerMap: removing container" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerName="cleanup" Nov 01 22:45:44.937062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937075 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937084 6466 cpu_manager.go:414] "containerMap: removing container" podUID="a1e35591-1c5a-485f-aea5-064e48802a9b" containerName="payload-processing" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937089 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e35591-1c5a-485f-aea5-064e48802a9b" containerName="payload-processing" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937098 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937102 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937218 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937226 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" containerName="istio-proxy" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937233 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937239 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937246 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e35591-1c5a-485f-aea5-064e48802a9b" containerName="payload-processing" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937253 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937259 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937265 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937269 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937274 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937281 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937286 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937291 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937298 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8221bb-bcff-4e2a-af5c-76f36b0a6477" containerName="istio-proxy" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937305 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937310 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937315 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="816a7bec-ff31-49a0-9b0c-0112e73b1a5a" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937322 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937327 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b08c73-7108-40f6-b0e0-456446484085" containerName="payload-pre-processing" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937334 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937341 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937346 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937352 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="14785ed4-8e76-4509-9712-0026d2e80b74" containerName="cleanup" Nov 01 22:45:44.938186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.937357 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4f2016-b4c5-45c7-a3cf-2772a27a8f7e" containerName="cleanup" Nov 01 22:45:44.944958 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.944933 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:44.954090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.953312 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9717e2-openshift-default-dockercfg-nbxqz\"" Nov 01 22:45:44.995801 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:44.988693 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt"] Nov 01 22:45:45.028131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.025533 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-istio-token\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.028131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.025630 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-socket\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.028131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.025691 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07b617c-2d6b-4814-af51-406639a9b920-istio-podinfo\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.028131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.025751 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07b617c-2d6b-4814-af51-406639a9b920-istiod-ca-cert\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.028131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.025820 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-credential-socket\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.028131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.025961 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-data\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.028131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.025988 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7b2\" (UniqueName: \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-kube-api-access-hw7b2\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.028131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.026051 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-certs\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.028131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.026099 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-envoy\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.137362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.137321 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-istio-token\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.137571 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.137402 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-socket\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.137571 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.137455 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07b617c-2d6b-4814-af51-406639a9b920-istio-podinfo\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.137571 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.137506 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07b617c-2d6b-4814-af51-406639a9b920-istiod-ca-cert\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.137571 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.137567 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-credential-socket\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.137793 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.137684 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-data\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.137793 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.137710 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7b2\" (UniqueName: \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-kube-api-access-hw7b2\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.137793 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.137776 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-certs\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.137940 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.137824 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-envoy\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.161655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.161609 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07b617c-2d6b-4814-af51-406639a9b920-istiod-ca-cert\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.162863 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.162295 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-socket\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.165329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.163362 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-certs\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.165329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.163616 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-credential-socket\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.165842 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.165769 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-data\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.189259 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.187532 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-envoy\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.216245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.213758 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07b617c-2d6b-4814-af51-406639a9b920-istio-podinfo\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.261731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.261679 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7b2\" (UniqueName: \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-kube-api-access-hw7b2\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.288093 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.288067 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-istio-token\") pod \"e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.484049 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.483927 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" Nov 01 22:45:45.484824 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.484036 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9" event={"ID":"1f1280b1-f9ea-489c-8b06-f595fae26903","Type":"ContainerDied","Data":"9462b93b3b1ad5e3b611a3eb579647f50493ebeaa905cc1d7d1dc9b57b802db5"} Nov 01 22:45:45.484824 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.484567 6466 scope.go:122] "RemoveContainer" containerID="739cbf8f725203790b4529424cebe4cfda23a0a58c723ae12ec61a816f04724b" Nov 01 22:45:45.516874 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.516843 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9"] Nov 01 22:45:45.524727 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.524704 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-10a1d9-openshift-default-8487d546d5-ggcx9"] Nov 01 22:45:45.584117 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.584073 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:45.827498 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.827349 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn"] Nov 01 22:45:45.895082 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:45:45.888321 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07b617c_2d6b_4814_af51_406639a9b920.slice/crio-88b33738128128677dcb65e5e77058f3a5f7efddfe0b72807f9b23c2646cd78b WatchSource:0}: Error finding container 88b33738128128677dcb65e5e77058f3a5f7efddfe0b72807f9b23c2646cd78b: Status 404 returned error can't find the container with id 88b33738128128677dcb65e5e77058f3a5f7efddfe0b72807f9b23c2646cd78b Nov 01 22:45:45.895082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.890575 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt"] Nov 01 22:45:45.895082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.891721 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:45:45.895082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.891781 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:45:45.895082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:45.891807 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:45:46.439895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:46.439857 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1280b1-f9ea-489c-8b06-f595fae26903" path="/var/lib/kubelet/pods/1f1280b1-f9ea-489c-8b06-f595fae26903/volumes" Nov 01 22:45:46.503477 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:46.503436 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" event={"ID":"a07b617c-2d6b-4814-af51-406639a9b920","Type":"ContainerStarted","Data":"13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23"} Nov 01 22:45:46.504009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:46.503508 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" event={"ID":"a07b617c-2d6b-4814-af51-406639a9b920","Type":"ContainerStarted","Data":"88b33738128128677dcb65e5e77058f3a5f7efddfe0b72807f9b23c2646cd78b"} Nov 01 22:45:46.542594 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:46.541797 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" podStartSLOduration=2.5417769740000002 podStartE2EDuration="2.541776974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:46.534821763 +0000 UTC m=+1350.319260685" watchObservedRunningTime="2026-11-01 22:45:46.541776974 +0000 UTC m=+1350.326215897" Nov 01 22:45:46.584599 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:46.584557 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:46.586302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:46.586269 6466 patch_prober.go:29] interesting pod/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 22:45:46.586438 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:46.586331 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 22:45:47.105143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.101848 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l"] Nov 01 22:45:47.105143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.102613 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" podUID="73cb80d1-5953-4b69-9c6b-51eef06e8336" containerName="maas-api" containerID="cri-o://57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1" gracePeriod=30 Nov 01 22:45:47.434280 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.434246 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:45:47.515303 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.515265 6466 generic.go:356] "Generic (PLEG): container finished" podID="73cb80d1-5953-4b69-9c6b-51eef06e8336" containerID="57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1" exitCode=0 Nov 01 22:45:47.515699 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.515325 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" Nov 01 22:45:47.515699 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.515349 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" event={"ID":"73cb80d1-5953-4b69-9c6b-51eef06e8336","Type":"ContainerDied","Data":"57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1"} Nov 01 22:45:47.515699 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.515404 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l" event={"ID":"73cb80d1-5953-4b69-9c6b-51eef06e8336","Type":"ContainerDied","Data":"39e2f14dce4fbab84d6d3c4b7e7db90fcfd2f85c43b5e92c4a2e88e8b71e7a3b"} Nov 01 22:45:47.515699 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.515430 6466 scope.go:122] "RemoveContainer" containerID="57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1" Nov 01 22:45:47.531303 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.531277 6466 scope.go:122] "RemoveContainer" containerID="57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1" Nov 01 22:45:47.531681 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:47.531650 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1\": container with ID starting with 57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1 not found: ID does not exist" containerID="57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1" Nov 01 22:45:47.531778 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.531688 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1"} err="failed to get container status \"57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1\": rpc error: code = NotFound desc = could not find container \"57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1\": container with ID starting with 57c5d70e9652f44ca6673b3c7a675f8903fc053d80e186af5f2ba57137fdc7c1 not found: ID does not exist" Nov 01 22:45:47.577610 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.577578 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-metrics-certs\") pod \"73cb80d1-5953-4b69-9c6b-51eef06e8336\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " Nov 01 22:45:47.577764 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.577695 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73cb80d1-5953-4b69-9c6b-51eef06e8336-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73cb80d1-5953-4b69-9c6b-51eef06e8336-openshift-service-ca\") pod \"73cb80d1-5953-4b69-9c6b-51eef06e8336\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " Nov 01 22:45:47.577764 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.577728 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-maas-api-tls\") pod \"73cb80d1-5953-4b69-9c6b-51eef06e8336\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " Nov 01 22:45:47.577858 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.577800 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73cb80d1-5953-4b69-9c6b-51eef06e8336-kube-api-access-qx6t5\" (UniqueName: \"kubernetes.io/projected/73cb80d1-5953-4b69-9c6b-51eef06e8336-kube-api-access-qx6t5\") pod \"73cb80d1-5953-4b69-9c6b-51eef06e8336\" (UID: \"73cb80d1-5953-4b69-9c6b-51eef06e8336\") " Nov 01 22:45:47.578136 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.578074 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73cb80d1-5953-4b69-9c6b-51eef06e8336-openshift-service-ca" pod "73cb80d1-5953-4b69-9c6b-51eef06e8336" (UID: "73cb80d1-5953-4b69-9c6b-51eef06e8336"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:47.578995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.578969 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73cb80d1-5953-4b69-9c6b-51eef06e8336-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:47.580393 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.580362 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73cb80d1-5953-4b69-9c6b-51eef06e8336-kube-api-access-qx6t5" pod "73cb80d1-5953-4b69-9c6b-51eef06e8336" (UID: "73cb80d1-5953-4b69-9c6b-51eef06e8336"). InnerVolumeSpecName "kube-api-access-qx6t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:47.580489 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.580462 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-maas-api-tls" pod "73cb80d1-5953-4b69-9c6b-51eef06e8336" (UID: "73cb80d1-5953-4b69-9c6b-51eef06e8336"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:47.580702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.580680 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-metrics-certs" pod "73cb80d1-5953-4b69-9c6b-51eef06e8336" (UID: "73cb80d1-5953-4b69-9c6b-51eef06e8336"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:47.589633 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.589611 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:47.681581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.681451 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:47.681581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.681489 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx6t5\" (UniqueName: \"kubernetes.io/projected/73cb80d1-5953-4b69-9c6b-51eef06e8336-kube-api-access-qx6t5\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:47.681581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.681503 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cb80d1-5953-4b69-9c6b-51eef06e8336-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:47.951887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:47.951159 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l"] Nov 01 22:45:48.115261 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:48.093307 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b42cc4-75b59d64c5-kvr5l"] Nov 01 22:45:48.343531 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:48.343446 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73cb80d1-5953-4b69-9c6b-51eef06e8336" path="/var/lib/kubelet/pods/73cb80d1-5953-4b69-9c6b-51eef06e8336/volumes" Nov 01 22:45:48.533798 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:48.533422 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:48.537799 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:48.535418 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:45:49.526655 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:49.526583 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:45:50.319740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:50.319697 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t"] Nov 01 22:45:50.472211 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:50.472159 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" podUID="aa72766e-f4b6-4ddd-9c8d-ad30edb83638" containerName="maas-api" containerID="cri-o://8b3854168aca8a06c48fee831d271b7b6f02dcb6eff43467cef9ae755ddc4dc8" gracePeriod=30 Nov 01 22:45:50.687891 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:50.687839 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:50.851055 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:50.845180 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" podUID="69034553-3a33-4319-b857-6fed9c037c1a" containerName="payload-pre-processing" containerID="cri-o://c6c091869d2ba4563d6f4602e059b88a3e7564b97b47cdc1a1aae1ef6266533d" gracePeriod=25 Nov 01 22:45:51.079194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:51.049377 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:51.088162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:51.055451 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:51.221604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:51.212396 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f"] Nov 01 22:45:51.443423 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:51.443364 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:51.690354 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:51.690297 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:51.692087 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:51.692065 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:45:52.099231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.090947 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:45:52.099231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.091322 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 01 22:45:52.120261 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.120056 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:45:52.120261 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.120160 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:45:52.217158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.141233 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt"] Nov 01 22:45:52.217158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.142486 6466 cpu_manager.go:414] "containerMap: removing container" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerName="cleanup" Nov 01 22:45:52.217158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.142511 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerName="cleanup" Nov 01 22:45:52.217158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.142546 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerName="cleanup" Nov 01 22:45:52.217158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.142554 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d680f9ed-27fd-44d0-a76c-1979761388a0" containerName="cleanup" Nov 01 22:45:52.217158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.142639 6466 cpu_manager.go:414] "containerMap: removing container" podUID="73cb80d1-5953-4b69-9c6b-51eef06e8336" containerName="maas-api" Nov 01 22:45:52.217158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.142649 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73cb80d1-5953-4b69-9c6b-51eef06e8336" containerName="maas-api" Nov 01 22:45:52.217158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.142938 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9d81f5-1e85-41b2-9eac-c43ccfb98773" containerName="cleanup" Nov 01 22:45:52.217158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.142961 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6d8cdd-7079-4eb3-8701-d26549713a55" containerName="cleanup" Nov 01 22:45:52.217158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.142977 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7562a4b4-c136-4cd1-98c5-aa27d2ec9cb7" containerName="cleanup" Nov 01 22:45:52.217158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.142993 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="73cb80d1-5953-4b69-9c6b-51eef06e8336" containerName="maas-api" Nov 01 22:45:52.411554 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.411520 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:45:52.551168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.500747 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9717e2\"" Nov 01 22:45:52.774186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.731092 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:45:52.774186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.734164 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdzzv\" (UniqueName: \"kubernetes.io/projected/54c5a94c-c2f6-492c-920a-efb24d34a409-kube-api-access-qdzzv\") pod \"payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt\" (UID: \"54c5a94c-c2f6-492c-920a-efb24d34a409\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:45:52.778129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.774638 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c5a94c-c2f6-492c-920a-efb24d34a409-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt\" (UID: \"54c5a94c-c2f6-492c-920a-efb24d34a409\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:45:52.909304 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:45:52.908982 6466 logging.go:55] [core] [Channel #2922 SubChannel #2923]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.102:50052", ServerName: "10.134.0.102:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.102:50052: connect: connection refused" Nov 01 22:45:52.966134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.951029 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c5a94c-c2f6-492c-920a-efb24d34a409-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt\" (UID: \"54c5a94c-c2f6-492c-920a-efb24d34a409\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:45:52.966134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:52.951315 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdzzv\" (UniqueName: \"kubernetes.io/projected/54c5a94c-c2f6-492c-920a-efb24d34a409-kube-api-access-qdzzv\") pod \"payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt\" (UID: \"54c5a94c-c2f6-492c-920a-efb24d34a409\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:45:53.030626 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:53.030542 6466 generic.go:356] "Generic (PLEG): container finished" podID="69034553-3a33-4319-b857-6fed9c037c1a" containerID="c6c091869d2ba4563d6f4602e059b88a3e7564b97b47cdc1a1aae1ef6266533d" exitCode=0 Nov 01 22:45:53.089061 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:53.089023 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c5a94c-c2f6-492c-920a-efb24d34a409-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt\" (UID: \"54c5a94c-c2f6-492c-920a-efb24d34a409\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:45:53.102276 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:53.102238 6466 generic.go:356] "Generic (PLEG): container finished" podID="aa72766e-f4b6-4ddd-9c8d-ad30edb83638" containerID="8b3854168aca8a06c48fee831d271b7b6f02dcb6eff43467cef9ae755ddc4dc8" exitCode=0 Nov 01 22:45:53.121527 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:53.121486 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:53.133944 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:53.133434 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:53.309271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:53.307028 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdzzv\" (UniqueName: \"kubernetes.io/projected/54c5a94c-c2f6-492c-920a-efb24d34a409-kube-api-access-qdzzv\") pod \"payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt\" (UID: \"54c5a94c-c2f6-492c-920a-efb24d34a409\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:45:53.597908 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:53.597807 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:45:53.914915 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:45:53.709948 6466 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.331s" Nov 01 22:45:53.915251 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:53.915227 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh"] Nov 01 22:45:53.948955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:53.842144 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:53.975132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:53.970079 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" podUID="69034553-3a33-4319-b857-6fed9c037c1a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.102:50052\" within 1s: context deadline exceeded" Nov 01 22:45:54.016131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:54.010421 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" event={"ID":"69034553-3a33-4319-b857-6fed9c037c1a","Type":"ContainerDied","Data":"c6c091869d2ba4563d6f4602e059b88a3e7564b97b47cdc1a1aae1ef6266533d"} Nov 01 22:45:54.042133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:54.029209 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh"] Nov 01 22:45:54.067133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:54.042405 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt"] Nov 01 22:45:54.067133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:54.059217 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" event={"ID":"aa72766e-f4b6-4ddd-9c8d-ad30edb83638","Type":"ContainerDied","Data":"8b3854168aca8a06c48fee831d271b7b6f02dcb6eff43467cef9ae755ddc4dc8"} Nov 01 22:45:54.067133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:54.059313 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:45:54.067133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:54.011454 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:45:54.221188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:54.152195 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:45:54.506257 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:54.500659 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:54.773898 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:54.773818 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:45:55.019277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.011350 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 22:45:55.159188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.130818 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:55.284897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.284862 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ksr\" (UniqueName: \"kubernetes.io/projected/8918ad96-0491-41aa-9a97-b67b78236470-kube-api-access-x9ksr\") pod \"payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh\" (UID: \"8918ad96-0491-41aa-9a97-b67b78236470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:45:55.296547 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.294791 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7klkb" podUID="66937625-7220-44c9-b11a-dbc627d0219c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:55.315472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.315438 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8918ad96-0491-41aa-9a97-b67b78236470-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh\" (UID: \"8918ad96-0491-41aa-9a97-b67b78236470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:45:55.329845 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.329808 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 01 22:45:55.339348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.339322 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:45:55.397799 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.397759 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" Nov 01 22:45:55.435174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.425646 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69034553-3a33-4319-b857-6fed9c037c1a-kube-api-access-5qdzm\" (UniqueName: \"kubernetes.io/projected/69034553-3a33-4319-b857-6fed9c037c1a-kube-api-access-5qdzm\") pod \"69034553-3a33-4319-b857-6fed9c037c1a\" (UID: \"69034553-3a33-4319-b857-6fed9c037c1a\") " Nov 01 22:45:55.435174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.425701 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-maas-api-tls\") pod \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " Nov 01 22:45:55.435174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.425922 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-metrics-certs\") pod \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " Nov 01 22:45:55.435174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.426081 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-kube-api-access-cqrl2\" (UniqueName: \"kubernetes.io/projected/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-kube-api-access-cqrl2\") pod \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " Nov 01 22:45:55.435174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.426652 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69034553-3a33-4319-b857-6fed9c037c1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69034553-3a33-4319-b857-6fed9c037c1a-plugins-config-volume\") pod \"69034553-3a33-4319-b857-6fed9c037c1a\" (UID: \"69034553-3a33-4319-b857-6fed9c037c1a\") " Nov 01 22:45:55.435174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.426699 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-openshift-service-ca\") pod \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\" (UID: \"aa72766e-f4b6-4ddd-9c8d-ad30edb83638\") " Nov 01 22:45:55.435174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.428486 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-openshift-service-ca" pod "aa72766e-f4b6-4ddd-9c8d-ad30edb83638" (UID: "aa72766e-f4b6-4ddd-9c8d-ad30edb83638"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:55.435174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.432287 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8918ad96-0491-41aa-9a97-b67b78236470-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh\" (UID: \"8918ad96-0491-41aa-9a97-b67b78236470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:45:55.435174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.432530 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ksr\" (UniqueName: \"kubernetes.io/projected/8918ad96-0491-41aa-9a97-b67b78236470-kube-api-access-x9ksr\") pod \"payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh\" (UID: \"8918ad96-0491-41aa-9a97-b67b78236470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:45:55.435174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.432897 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:55.435174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.434432 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69034553-3a33-4319-b857-6fed9c037c1a-plugins-config-volume" pod "69034553-3a33-4319-b857-6fed9c037c1a" (UID: "69034553-3a33-4319-b857-6fed9c037c1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:55.445168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.442027 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8918ad96-0491-41aa-9a97-b67b78236470-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh\" (UID: \"8918ad96-0491-41aa-9a97-b67b78236470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:45:55.445168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.443989 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:45:55.445168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.444023 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" event={"ID":"aa72766e-f4b6-4ddd-9c8d-ad30edb83638","Type":"ContainerDied","Data":"788ae3e93c4bea38769a0af4fdfe24c249bc0ff04ae53a2b2f85133a1cb061e1"} Nov 01 22:45:55.445168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.444149 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:45:55.445168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.444216 6466 scope.go:122] "RemoveContainer" containerID="8b3854168aca8a06c48fee831d271b7b6f02dcb6eff43467cef9ae755ddc4dc8" Nov 01 22:45:55.558876 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.551050 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-kube-api-access-cqrl2" pod "aa72766e-f4b6-4ddd-9c8d-ad30edb83638" (UID: "aa72766e-f4b6-4ddd-9c8d-ad30edb83638"). InnerVolumeSpecName "kube-api-access-cqrl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:55.595918 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.582389 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqrl2\" (UniqueName: \"kubernetes.io/projected/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-kube-api-access-cqrl2\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:55.596336 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.596308 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69034553-3a33-4319-b857-6fed9c037c1a-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:55.613659 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.613620 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-metrics-certs" pod "aa72766e-f4b6-4ddd-9c8d-ad30edb83638" (UID: "aa72766e-f4b6-4ddd-9c8d-ad30edb83638"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:55.615564 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.615535 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-maas-api-tls" pod "aa72766e-f4b6-4ddd-9c8d-ad30edb83638" (UID: "aa72766e-f4b6-4ddd-9c8d-ad30edb83638"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:55.617825 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.617800 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ksr\" (UniqueName: \"kubernetes.io/projected/8918ad96-0491-41aa-9a97-b67b78236470-kube-api-access-x9ksr\") pod \"payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh\" (UID: \"8918ad96-0491-41aa-9a97-b67b78236470\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:45:55.629857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.629824 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69034553-3a33-4319-b857-6fed9c037c1a-kube-api-access-5qdzm" pod "69034553-3a33-4319-b857-6fed9c037c1a" (UID: "69034553-3a33-4319-b857-6fed9c037c1a"). InnerVolumeSpecName "kube-api-access-5qdzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:55.747189 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.727274 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qdzm\" (UniqueName: \"kubernetes.io/projected/69034553-3a33-4319-b857-6fed9c037c1a-kube-api-access-5qdzm\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:55.827704 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.827672 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:45:55.828308 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.827705 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:55.828431 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.828336 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa72766e-f4b6-4ddd-9c8d-ad30edb83638-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:55.914185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.914150 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t"] Nov 01 22:45:55.933251 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:55.932648 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t"] Nov 01 22:45:56.175728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.167279 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt"] Nov 01 22:45:56.175728 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:45:56.170302 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c5a94c_c2f6_492c_920a_efb24d34a409.slice/crio-38d9c124603abaa306f3561456831122a70f9b3be403a7c7183d47c3ba83751c WatchSource:0}: Error finding container 38d9c124603abaa306f3561456831122a70f9b3be403a7c7183d47c3ba83751c: Status 404 returned error can't find the container with id 38d9c124603abaa306f3561456831122a70f9b3be403a7c7183d47c3ba83751c Nov 01 22:45:56.178284 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.178257 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-0fd159-f589d8bf7-chq2t" podUID="aa72766e-f4b6-4ddd-9c8d-ad30edb83638" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.104:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:56.240046 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.236348 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" podUID="71276961-b300-4137-8f01-eb480fa73659" containerName="payload-processing" containerID="cri-o://307f758967e7ffdb9ef1abdc9be1be04e7bb2af3d1df81ea6bffda8958222d63" gracePeriod=25 Nov 01 22:45:56.328775 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.328739 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh"] Nov 01 22:45:56.330332 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:45:56.330300 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8918ad96_0491_41aa_9a97_b67b78236470.slice/crio-6bc7a586fa7d207fe2c73354c28b74d7a6bd3b180fb22155c6d6c19f224fdcef WatchSource:0}: Error finding container 6bc7a586fa7d207fe2c73354c28b74d7a6bd3b180fb22155c6d6c19f224fdcef: Status 404 returned error can't find the container with id 6bc7a586fa7d207fe2c73354c28b74d7a6bd3b180fb22155c6d6c19f224fdcef Nov 01 22:45:56.369228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.369203 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa72766e-f4b6-4ddd-9c8d-ad30edb83638" path="/var/lib/kubelet/pods/aa72766e-f4b6-4ddd-9c8d-ad30edb83638/volumes" Nov 01 22:45:56.407231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.407200 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" event={"ID":"8918ad96-0491-41aa-9a97-b67b78236470","Type":"ContainerStarted","Data":"6bc7a586fa7d207fe2c73354c28b74d7a6bd3b180fb22155c6d6c19f224fdcef"} Nov 01 22:45:56.408964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.408938 6466 generic.go:356] "Generic (PLEG): container finished" podID="71276961-b300-4137-8f01-eb480fa73659" containerID="307f758967e7ffdb9ef1abdc9be1be04e7bb2af3d1df81ea6bffda8958222d63" exitCode=0 Nov 01 22:45:56.409196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.409174 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" event={"ID":"71276961-b300-4137-8f01-eb480fa73659","Type":"ContainerDied","Data":"307f758967e7ffdb9ef1abdc9be1be04e7bb2af3d1df81ea6bffda8958222d63"} Nov 01 22:45:56.413206 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.413142 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" event={"ID":"54c5a94c-c2f6-492c-920a-efb24d34a409","Type":"ContainerStarted","Data":"5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c"} Nov 01 22:45:56.413206 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.413184 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" event={"ID":"54c5a94c-c2f6-492c-920a-efb24d34a409","Type":"ContainerStarted","Data":"38d9c124603abaa306f3561456831122a70f9b3be403a7c7183d47c3ba83751c"} Nov 01 22:45:56.413342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.413289 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:45:56.416738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.416714 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" Nov 01 22:45:56.416859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.416733 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn" event={"ID":"69034553-3a33-4319-b857-6fed9c037c1a","Type":"ContainerDied","Data":"b13e3ad5b8d999306d745192b094a06e84808e4ba235051613055c37bf176adc"} Nov 01 22:45:56.416859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.416791 6466 scope.go:122] "RemoveContainer" containerID="c6c091869d2ba4563d6f4602e059b88a3e7564b97b47cdc1a1aae1ef6266533d" Nov 01 22:45:56.451182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.450441 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" podStartSLOduration=4.45042052 podStartE2EDuration="4.45042052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:56.435082195 +0000 UTC m=+1360.219521116" watchObservedRunningTime="2026-11-01 22:45:56.45042052 +0000 UTC m=+1360.234859442" Nov 01 22:45:56.477296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.477268 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn"] Nov 01 22:45:56.494716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.494684 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0fd159-6cc84466c5-65jzn"] Nov 01 22:45:56.721285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.721247 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:45:56.835974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.835935 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71276961-b300-4137-8f01-eb480fa73659-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71276961-b300-4137-8f01-eb480fa73659-plugins-config-volume\") pod \"71276961-b300-4137-8f01-eb480fa73659\" (UID: \"71276961-b300-4137-8f01-eb480fa73659\") " Nov 01 22:45:56.861554 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.861526 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71276961-b300-4137-8f01-eb480fa73659-kube-api-access-c2zgf\" (UniqueName: \"kubernetes.io/projected/71276961-b300-4137-8f01-eb480fa73659-kube-api-access-c2zgf\") pod \"71276961-b300-4137-8f01-eb480fa73659\" (UID: \"71276961-b300-4137-8f01-eb480fa73659\") " Nov 01 22:45:56.890149 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.861342 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71276961-b300-4137-8f01-eb480fa73659-plugins-config-volume" pod "71276961-b300-4137-8f01-eb480fa73659" (UID: "71276961-b300-4137-8f01-eb480fa73659"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:56.890149 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:56.887698 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71276961-b300-4137-8f01-eb480fa73659-kube-api-access-c2zgf" pod "71276961-b300-4137-8f01-eb480fa73659" (UID: "71276961-b300-4137-8f01-eb480fa73659"). InnerVolumeSpecName "kube-api-access-c2zgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:57.003689 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:57.003596 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71276961-b300-4137-8f01-eb480fa73659-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:57.003689 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:57.003641 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2zgf\" (UniqueName: \"kubernetes.io/projected/71276961-b300-4137-8f01-eb480fa73659-kube-api-access-c2zgf\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:57.431568 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:57.431533 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" event={"ID":"8918ad96-0491-41aa-9a97-b67b78236470","Type":"ContainerStarted","Data":"09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40"} Nov 01 22:45:57.431756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:57.431627 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:45:57.432973 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:57.432948 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" Nov 01 22:45:57.433101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:57.432974 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f" event={"ID":"71276961-b300-4137-8f01-eb480fa73659","Type":"ContainerDied","Data":"920178cee1784b3a84aa856040aabc1046f2800b93c5de542c41e81cbfcf5135"} Nov 01 22:45:57.433101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:57.433029 6466 scope.go:122] "RemoveContainer" containerID="307f758967e7ffdb9ef1abdc9be1be04e7bb2af3d1df81ea6bffda8958222d63" Nov 01 22:45:57.456650 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:57.456600 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" podStartSLOduration=5.456587521 podStartE2EDuration="5.456587521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:57.455618737 +0000 UTC m=+1361.240057663" watchObservedRunningTime="2026-11-01 22:45:57.456587521 +0000 UTC m=+1361.241026441" Nov 01 22:45:57.487029 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:57.486994 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f"] Nov 01 22:45:57.494522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:57.494491 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-41b8f5-597dfc7849-ml47f"] Nov 01 22:45:58.358802 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:58.358758 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69034553-3a33-4319-b857-6fed9c037c1a" path="/var/lib/kubelet/pods/69034553-3a33-4319-b857-6fed9c037c1a/volumes" Nov 01 22:45:58.359520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:58.359499 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71276961-b300-4137-8f01-eb480fa73659" path="/var/lib/kubelet/pods/71276961-b300-4137-8f01-eb480fa73659/volumes" Nov 01 22:45:58.619413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:58.619332 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk"] Nov 01 22:45:58.619600 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:58.619577 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" containerID="cri-o://2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee" gracePeriod=30 Nov 01 22:45:58.628224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:58.628196 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8"] Nov 01 22:45:58.628412 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:45:58.628392 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" containerID="cri-o://1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53" gracePeriod=30 Nov 01 22:46:02.184232 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.184194 6466 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:46:02.186335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.184275 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:46:02.277532 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.277260 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": read tcp 10.134.0.2:56180->10.134.0.148:15021: read: connection reset by peer" start-of-body= Nov 01 22:46:02.277532 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.277347 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": read tcp 10.134.0.2:56180->10.134.0.148:15021: read: connection reset by peer" Nov 01 22:46:02.555528 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.555431 6466 generic.go:356] "Generic (PLEG): container finished" podID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerID="6a59eaf29dda5b74b21b0d6917afd14563d62b6b4655be6076ccb83da3b7cd4f" exitCode=0 Nov 01 22:46:02.555684 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.555524 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" event={"ID":"948c76bd-1965-496d-9c0d-cbb6bc453ffc","Type":"ContainerDied","Data":"6a59eaf29dda5b74b21b0d6917afd14563d62b6b4655be6076ccb83da3b7cd4f"} Nov 01 22:46:02.556170 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.556149 6466 scope.go:122] "RemoveContainer" containerID="6a59eaf29dda5b74b21b0d6917afd14563d62b6b4655be6076ccb83da3b7cd4f" Nov 01 22:46:02.558530 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.558492 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:46:02.558616 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.558586 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:46:02.558661 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.558637 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:46:02.569474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.569444 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57"] Nov 01 22:46:02.569732 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:02.569706 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" containerID="cri-o://85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba" gracePeriod=30 Nov 01 22:46:03.567325 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:03.567289 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" event={"ID":"948c76bd-1965-496d-9c0d-cbb6bc453ffc","Type":"ContainerStarted","Data":"58b86e24e7f8bf263c4146b6943152b17d9f8298655d035f4b39ec8b34711527"} Nov 01 22:46:03.779586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:03.779550 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:46:03.779917 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:03.779888 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 22:46:03.780000 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:03.779968 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 22:46:03.987770 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:03.987690 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:46:04.173198 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.173171 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:46:04.183228 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183196 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-token\" (UniqueName: \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-token\") pod \"d52a7224-6cd7-44ea-9bf5-d16246da3250\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " Nov 01 22:46:04.183426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183247 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-data\") pod \"d52a7224-6cd7-44ea-9bf5-d16246da3250\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " Nov 01 22:46:04.183426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183348 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-kube-api-access-vk8rt\" (UniqueName: \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-kube-api-access-vk8rt\") pod \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " Nov 01 22:46:04.183426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183398 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-podinfo\") pod \"d52a7224-6cd7-44ea-9bf5-d16246da3250\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " Nov 01 22:46:04.183604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183439 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-certs\") pod \"d52a7224-6cd7-44ea-9bf5-d16246da3250\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " Nov 01 22:46:04.183604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183493 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-data\") pod \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " Nov 01 22:46:04.183604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183532 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-socket\") pod \"d52a7224-6cd7-44ea-9bf5-d16246da3250\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " Nov 01 22:46:04.183751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183612 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-credential-socket\") pod \"d52a7224-6cd7-44ea-9bf5-d16246da3250\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " Nov 01 22:46:04.183751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183663 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52a7224-6cd7-44ea-9bf5-d16246da3250-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52a7224-6cd7-44ea-9bf5-d16246da3250-istiod-ca-cert\") pod \"d52a7224-6cd7-44ea-9bf5-d16246da3250\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " Nov 01 22:46:04.183751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183692 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-socket\") pod \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " Nov 01 22:46:04.183888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183782 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-credential-socket\") pod \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " Nov 01 22:46:04.183986 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.183966 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-token\" (UniqueName: \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-token\") pod \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " Nov 01 22:46:04.184210 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.184190 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-certs\") pod \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " Nov 01 22:46:04.184421 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.184403 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-kube-api-access-6zc7h\" (UniqueName: \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-kube-api-access-6zc7h\") pod \"d52a7224-6cd7-44ea-9bf5-d16246da3250\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " Nov 01 22:46:04.184570 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.184554 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-envoy\") pod \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " Nov 01 22:46:04.184718 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.184700 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-envoy\") pod \"d52a7224-6cd7-44ea-9bf5-d16246da3250\" (UID: \"d52a7224-6cd7-44ea-9bf5-d16246da3250\") " Nov 01 22:46:04.184876 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.184861 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istiod-ca-cert\") pod \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " Nov 01 22:46:04.185037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.185020 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-podinfo\") pod \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\" (UID: \"a7e7ebcf-30e6-4c53-836e-07acb021ae73\") " Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.184259 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-data" pod "d52a7224-6cd7-44ea-9bf5-d16246da3250" (UID: "d52a7224-6cd7-44ea-9bf5-d16246da3250"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.184486 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-data" pod "a7e7ebcf-30e6-4c53-836e-07acb021ae73" (UID: "a7e7ebcf-30e6-4c53-836e-07acb021ae73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.187832 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-kube-api-access-6zc7h" pod "d52a7224-6cd7-44ea-9bf5-d16246da3250" (UID: "d52a7224-6cd7-44ea-9bf5-d16246da3250"). InnerVolumeSpecName "kube-api-access-6zc7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.184648 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-credential-socket" pod "a7e7ebcf-30e6-4c53-836e-07acb021ae73" (UID: "a7e7ebcf-30e6-4c53-836e-07acb021ae73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.184943 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52a7224-6cd7-44ea-9bf5-d16246da3250-istiod-ca-cert" pod "d52a7224-6cd7-44ea-9bf5-d16246da3250" (UID: "d52a7224-6cd7-44ea-9bf5-d16246da3250"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.185177 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-socket" pod "a7e7ebcf-30e6-4c53-836e-07acb021ae73" (UID: "a7e7ebcf-30e6-4c53-836e-07acb021ae73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.185669 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-certs" pod "d52a7224-6cd7-44ea-9bf5-d16246da3250" (UID: "d52a7224-6cd7-44ea-9bf5-d16246da3250"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.185714 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-socket" pod "d52a7224-6cd7-44ea-9bf5-d16246da3250" (UID: "d52a7224-6cd7-44ea-9bf5-d16246da3250"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.186037 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-credential-socket" pod "d52a7224-6cd7-44ea-9bf5-d16246da3250" (UID: "d52a7224-6cd7-44ea-9bf5-d16246da3250"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.186052 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istiod-ca-cert" pod "a7e7ebcf-30e6-4c53-836e-07acb021ae73" (UID: "a7e7ebcf-30e6-4c53-836e-07acb021ae73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.186324 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-certs" pod "a7e7ebcf-30e6-4c53-836e-07acb021ae73" (UID: "a7e7ebcf-30e6-4c53-836e-07acb021ae73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.187799 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-kube-api-access-vk8rt" pod "a7e7ebcf-30e6-4c53-836e-07acb021ae73" (UID: "a7e7ebcf-30e6-4c53-836e-07acb021ae73"). InnerVolumeSpecName "kube-api-access-vk8rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188488 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188512 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zc7h\" (UniqueName: \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-kube-api-access-6zc7h\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188525 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188538 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188551 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk8rt\" (UniqueName: \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-kube-api-access-vk8rt\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188563 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188577 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188588 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188601 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188616 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52a7224-6cd7-44ea-9bf5-d16246da3250-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188628 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.189723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.188640 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.191134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.190398 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-token" pod "a7e7ebcf-30e6-4c53-836e-07acb021ae73" (UID: "a7e7ebcf-30e6-4c53-836e-07acb021ae73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:04.191134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.190519 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-podinfo" pod "d52a7224-6cd7-44ea-9bf5-d16246da3250" (UID: "d52a7224-6cd7-44ea-9bf5-d16246da3250"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:04.191134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.190529 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-envoy" pod "a7e7ebcf-30e6-4c53-836e-07acb021ae73" (UID: "a7e7ebcf-30e6-4c53-836e-07acb021ae73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.191678 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.191434 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-token" pod "d52a7224-6cd7-44ea-9bf5-d16246da3250" (UID: "d52a7224-6cd7-44ea-9bf5-d16246da3250"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:04.191922 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.191898 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-envoy" pod "d52a7224-6cd7-44ea-9bf5-d16246da3250" (UID: "d52a7224-6cd7-44ea-9bf5-d16246da3250"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.192207 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.192186 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-podinfo" pod "a7e7ebcf-30e6-4c53-836e-07acb021ae73" (UID: "a7e7ebcf-30e6-4c53-836e-07acb021ae73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:04.224242 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.222298 6466 patch_prober.go:29] interesting pod/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:46:04.224242 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.222369 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:46:04.290719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.290681 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.290719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.290714 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.290719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.290727 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.290980 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.290742 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.290980 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.290755 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52a7224-6cd7-44ea-9bf5-d16246da3250-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.290980 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.290766 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e7ebcf-30e6-4c53-836e-07acb021ae73-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.576906 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.576872 6466 generic.go:356] "Generic (PLEG): container finished" podID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerID="1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53" exitCode=0 Nov 01 22:46:04.577329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.576984 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" Nov 01 22:46:04.577329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.577007 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" event={"ID":"d52a7224-6cd7-44ea-9bf5-d16246da3250","Type":"ContainerDied","Data":"1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53"} Nov 01 22:46:04.577329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.577063 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8" event={"ID":"d52a7224-6cd7-44ea-9bf5-d16246da3250","Type":"ContainerDied","Data":"56dca0a7fd3e338da4ba35245dedcd2118f0182d095d49a417538ed192c8ec95"} Nov 01 22:46:04.577329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.577088 6466 scope.go:122] "RemoveContainer" containerID="1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53" Nov 01 22:46:04.579009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.578986 6466 generic.go:356] "Generic (PLEG): container finished" podID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerID="2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee" exitCode=0 Nov 01 22:46:04.579176 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.579041 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" Nov 01 22:46:04.579176 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.579036 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" event={"ID":"a7e7ebcf-30e6-4c53-836e-07acb021ae73","Type":"ContainerDied","Data":"2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee"} Nov 01 22:46:04.579176 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.579096 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk" event={"ID":"a7e7ebcf-30e6-4c53-836e-07acb021ae73","Type":"ContainerDied","Data":"980b521a91fc401dbb0940596a502ff514ccccccac4541e430b542a5c64b85cf"} Nov 01 22:46:04.589396 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.589373 6466 scope.go:122] "RemoveContainer" containerID="46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91" Nov 01 22:46:04.608693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.608554 6466 scope.go:122] "RemoveContainer" containerID="1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53" Nov 01 22:46:04.608954 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:04.608936 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53\": container with ID starting with 1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53 not found: ID does not exist" containerID="1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53" Nov 01 22:46:04.609025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.608964 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53"} err="failed to get container status \"1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53\": rpc error: code = NotFound desc = could not find container \"1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53\": container with ID starting with 1050aa8e355cad88e94995668092ec3939f826449248c79a5fc5b6b9d44ddf53 not found: ID does not exist" Nov 01 22:46:04.609025 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.608985 6466 scope.go:122] "RemoveContainer" containerID="46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91" Nov 01 22:46:04.609296 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:04.609278 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91\": container with ID starting with 46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91 not found: ID does not exist" containerID="46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91" Nov 01 22:46:04.609366 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.609307 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91"} err="failed to get container status \"46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91\": rpc error: code = NotFound desc = could not find container \"46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91\": container with ID starting with 46fabca7816bbea7df04a3362a2399e266de6efa052d43a993a03e6a3c1a5b91 not found: ID does not exist" Nov 01 22:46:04.609366 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.609322 6466 scope.go:122] "RemoveContainer" containerID="2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee" Nov 01 22:46:04.612543 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.612517 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8"] Nov 01 22:46:04.618887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.618863 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-fwns8"] Nov 01 22:46:04.622082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.622065 6466 scope.go:122] "RemoveContainer" containerID="2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee" Nov 01 22:46:04.622445 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:04.622425 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee\": container with ID starting with 2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee not found: ID does not exist" containerID="2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee" Nov 01 22:46:04.622498 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.622454 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee"} err="failed to get container status \"2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee\": rpc error: code = NotFound desc = could not find container \"2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee\": container with ID starting with 2625a7bcd81c33f68f08ec0b0b073635a8dbc784ba62d8a0b09758e0a8d5d4ee not found: ID does not exist" Nov 01 22:46:04.634646 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.634610 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk"] Nov 01 22:46:04.667963 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.667928 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-cr4xk"] Nov 01 22:46:04.779335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.779296 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 22:46:04.779501 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:04.779378 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 22:46:05.779408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:05.779375 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 22:46:05.779823 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:05.779432 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 22:46:06.374655 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.374623 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" path="/var/lib/kubelet/pods/a7e7ebcf-30e6-4c53-836e-07acb021ae73/volumes" Nov 01 22:46:06.375214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.375196 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" path="/var/lib/kubelet/pods/d52a7224-6cd7-44ea-9bf5-d16246da3250/volumes" Nov 01 22:46:06.779702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.779667 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 22:46:06.780083 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.779721 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 22:46:06.905014 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.904974 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp"] Nov 01 22:46:06.906154 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906132 6466 cpu_manager.go:414] "containerMap: removing container" podUID="69034553-3a33-4319-b857-6fed9c037c1a" containerName="payload-pre-processing" Nov 01 22:46:06.906154 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906154 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69034553-3a33-4319-b857-6fed9c037c1a" containerName="payload-pre-processing" Nov 01 22:46:06.906347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906183 6466 cpu_manager.go:414] "containerMap: removing container" podUID="aa72766e-f4b6-4ddd-9c8d-ad30edb83638" containerName="maas-api" Nov 01 22:46:06.906347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906191 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa72766e-f4b6-4ddd-9c8d-ad30edb83638" containerName="maas-api" Nov 01 22:46:06.906347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906220 6466 cpu_manager.go:414] "containerMap: removing container" podUID="71276961-b300-4137-8f01-eb480fa73659" containerName="payload-processing" Nov 01 22:46:06.906347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906229 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71276961-b300-4137-8f01-eb480fa73659" containerName="payload-processing" Nov 01 22:46:06.906347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906268 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" Nov 01 22:46:06.906347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906275 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" Nov 01 22:46:06.906347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906303 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" Nov 01 22:46:06.906347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906311 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" Nov 01 22:46:06.906347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906326 6466 cpu_manager.go:414] "containerMap: removing container" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" Nov 01 22:46:06.906347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906333 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" Nov 01 22:46:06.906861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906504 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" Nov 01 22:46:06.906861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906525 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52a7224-6cd7-44ea-9bf5-d16246da3250" containerName="istio-proxy" Nov 01 22:46:06.906861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906534 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e7ebcf-30e6-4c53-836e-07acb021ae73" containerName="istio-proxy" Nov 01 22:46:06.906861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906551 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="69034553-3a33-4319-b857-6fed9c037c1a" containerName="payload-pre-processing" Nov 01 22:46:06.906861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906565 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa72766e-f4b6-4ddd-9c8d-ad30edb83638" containerName="maas-api" Nov 01 22:46:06.906861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.906577 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="71276961-b300-4137-8f01-eb480fa73659" containerName="payload-processing" Nov 01 22:46:06.913443 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.913414 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:06.928812 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.928790 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9717e2-serving-cert\"" Nov 01 22:46:06.967148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.962767 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-metrics-certs\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:06.967148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.962852 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-maas-api-tls\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:06.967148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.962944 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mn8z\" (UniqueName: \"kubernetes.io/projected/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-kube-api-access-2mn8z\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:06.967148 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.963002 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:06.969073 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:06.969036 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp"] Nov 01 22:46:07.086600 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.086510 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-metrics-certs\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:07.086895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.086876 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-maas-api-tls\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:07.087172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.087149 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mn8z\" (UniqueName: \"kubernetes.io/projected/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-kube-api-access-2mn8z\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:07.087440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.087409 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:07.088682 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.088650 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:07.096876 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.096579 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-maas-api-tls\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:07.097689 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.097662 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-metrics-certs\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:07.154751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.153818 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mn8z\" (UniqueName: \"kubernetes.io/projected/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-kube-api-access-2mn8z\") pod \"maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:07.232361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.232151 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:07.494814 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.494606 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:46:07.786135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.785554 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 22:46:07.786135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.785616 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 22:46:07.976071 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:07.976029 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp"] Nov 01 22:46:08.211150 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.211086 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:46:08.322182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322146 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-socket\") pod \"7d8ca67a-a7ed-4a7c-905e-f02763779799\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " Nov 01 22:46:08.322366 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322213 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-token\" (UniqueName: \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-token\") pod \"7d8ca67a-a7ed-4a7c-905e-f02763779799\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " Nov 01 22:46:08.322366 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322265 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-envoy\") pod \"7d8ca67a-a7ed-4a7c-905e-f02763779799\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " Nov 01 22:46:08.322475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322368 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-podinfo\") pod \"7d8ca67a-a7ed-4a7c-905e-f02763779799\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " Nov 01 22:46:08.322475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322430 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-socket" pod "7d8ca67a-a7ed-4a7c-905e-f02763779799" (UID: "7d8ca67a-a7ed-4a7c-905e-f02763779799"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:08.322587 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322507 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d8ca67a-a7ed-4a7c-905e-f02763779799-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d8ca67a-a7ed-4a7c-905e-f02763779799-istiod-ca-cert\") pod \"7d8ca67a-a7ed-4a7c-905e-f02763779799\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " Nov 01 22:46:08.322653 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322608 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-kube-api-access-f52hf\" (UniqueName: \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-kube-api-access-f52hf\") pod \"7d8ca67a-a7ed-4a7c-905e-f02763779799\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " Nov 01 22:46:08.322709 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322649 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-data\") pod \"7d8ca67a-a7ed-4a7c-905e-f02763779799\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " Nov 01 22:46:08.322709 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322678 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-credential-socket\") pod \"7d8ca67a-a7ed-4a7c-905e-f02763779799\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " Nov 01 22:46:08.322815 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322727 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-certs\") pod \"7d8ca67a-a7ed-4a7c-905e-f02763779799\" (UID: \"7d8ca67a-a7ed-4a7c-905e-f02763779799\") " Nov 01 22:46:08.322964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322923 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-credential-socket" pod "7d8ca67a-a7ed-4a7c-905e-f02763779799" (UID: "7d8ca67a-a7ed-4a7c-905e-f02763779799"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:08.322964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322945 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-data" pod "7d8ca67a-a7ed-4a7c-905e-f02763779799" (UID: "7d8ca67a-a7ed-4a7c-905e-f02763779799"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:08.322964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.322931 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8ca67a-a7ed-4a7c-905e-f02763779799-istiod-ca-cert" pod "7d8ca67a-a7ed-4a7c-905e-f02763779799" (UID: "7d8ca67a-a7ed-4a7c-905e-f02763779799"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:08.323248 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.323212 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-certs" pod "7d8ca67a-a7ed-4a7c-905e-f02763779799" (UID: "7d8ca67a-a7ed-4a7c-905e-f02763779799"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:08.323633 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.323610 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:08.323736 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.323638 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:08.323736 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.323653 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:08.323736 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.323669 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:08.323736 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.323684 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d8ca67a-a7ed-4a7c-905e-f02763779799-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:08.324741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.324718 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-envoy" pod "7d8ca67a-a7ed-4a7c-905e-f02763779799" (UID: "7d8ca67a-a7ed-4a7c-905e-f02763779799"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:08.324831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.324784 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-token" pod "7d8ca67a-a7ed-4a7c-905e-f02763779799" (UID: "7d8ca67a-a7ed-4a7c-905e-f02763779799"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:08.324831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.324803 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-podinfo" pod "7d8ca67a-a7ed-4a7c-905e-f02763779799" (UID: "7d8ca67a-a7ed-4a7c-905e-f02763779799"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:08.325192 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.325175 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-kube-api-access-f52hf" pod "7d8ca67a-a7ed-4a7c-905e-f02763779799" (UID: "7d8ca67a-a7ed-4a7c-905e-f02763779799"). InnerVolumeSpecName "kube-api-access-f52hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:08.425944 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.425914 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f52hf\" (UniqueName: \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-kube-api-access-f52hf\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:08.425944 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.425951 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:08.426103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.425968 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:08.426103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.425983 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d8ca67a-a7ed-4a7c-905e-f02763779799-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:08.473647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.473616 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:46:08.799331 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.799240 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 01 22:46:08.799331 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.799306 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 01 22:46:08.823129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.822981 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" event={"ID":"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231","Type":"ContainerStarted","Data":"3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c"} Nov 01 22:46:08.823129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.823059 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" event={"ID":"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231","Type":"ContainerStarted","Data":"84c37d1fd3c6659798acb04a518fde734214903c8813bc15bb3407630d4779fa"} Nov 01 22:46:08.823441 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.823190 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:08.824843 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.824818 6466 generic.go:356] "Generic (PLEG): container finished" podID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerID="85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba" exitCode=0 Nov 01 22:46:08.824955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.824848 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" event={"ID":"7d8ca67a-a7ed-4a7c-905e-f02763779799","Type":"ContainerDied","Data":"85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba"} Nov 01 22:46:08.824955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.824878 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" event={"ID":"7d8ca67a-a7ed-4a7c-905e-f02763779799","Type":"ContainerDied","Data":"1ecb87ed4185d436f2d3a290f6b680aef39759fdb8aadad1acf45b1784d8c4bd"} Nov 01 22:46:08.824955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.824899 6466 scope.go:122] "RemoveContainer" containerID="85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba" Nov 01 22:46:08.825057 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.825007 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57" Nov 01 22:46:08.867673 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.866255 6466 scope.go:122] "RemoveContainer" containerID="5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366" Nov 01 22:46:08.878779 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.878751 6466 scope.go:122] "RemoveContainer" containerID="85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba" Nov 01 22:46:08.879147 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:08.879090 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba\": container with ID starting with 85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba not found: ID does not exist" containerID="85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba" Nov 01 22:46:08.879231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.879146 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba"} err="failed to get container status \"85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba\": rpc error: code = NotFound desc = could not find container \"85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba\": container with ID starting with 85701dd53cd50034886acaea3378fb17498ac88aa9fddb959b515702e6dd3bba not found: ID does not exist" Nov 01 22:46:08.879231 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.879167 6466 scope.go:122] "RemoveContainer" containerID="5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366" Nov 01 22:46:08.879469 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:08.879449 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366\": container with ID starting with 5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366 not found: ID does not exist" containerID="5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366" Nov 01 22:46:08.879523 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.879480 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366"} err="failed to get container status \"5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366\": rpc error: code = NotFound desc = could not find container \"5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366\": container with ID starting with 5229a046bea2b79e97a515ffc20e0e40f1233cc4d496eb34892faa286ef8a366 not found: ID does not exist" Nov 01 22:46:08.898956 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.898911 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" podStartSLOduration=2.463978942 podStartE2EDuration="2.898896078s" totalImagesPullingTime="434.917136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:08.866594051 +0000 UTC m=+1372.651032973" watchObservedRunningTime="2026-11-01 22:46:08.898896078 +0000 UTC m=+1372.683335066" Nov 01 22:46:08.902138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.902117 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57"] Nov 01 22:46:08.909968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:08.909943 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0fd159-openshift-default-548c8fc57c-dgx57"] Nov 01 22:46:09.783660 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:09.783626 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:46:09.784152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:09.784127 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:46:09.784918 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:09.784902 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:46:10.435591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:10.435481 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" path="/var/lib/kubelet/pods/7d8ca67a-a7ed-4a7c-905e-f02763779799/volumes" Nov 01 22:46:11.494186 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:11.494153 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz"] Nov 01 22:46:13.260001 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:13.259233 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc"] Nov 01 22:46:13.270439 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:13.269909 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw"] Nov 01 22:46:14.899823 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:14.899791 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:46:15.557483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:15.557450 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx"] Nov 01 22:46:15.567544 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:15.567510 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs"] Nov 01 22:46:16.507135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.499189 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" podUID="22946ea8-86f3-4945-93a7-7e100bcf37a9" containerName="payload-pre-processing" containerID="cri-o://2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5" gracePeriod=25 Nov 01 22:46:16.832159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.832126 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:46:16.957690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.957656 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22946ea8-86f3-4945-93a7-7e100bcf37a9-kube-api-access-mc7m8\" (UniqueName: \"kubernetes.io/projected/22946ea8-86f3-4945-93a7-7e100bcf37a9-kube-api-access-mc7m8\") pod \"22946ea8-86f3-4945-93a7-7e100bcf37a9\" (UID: \"22946ea8-86f3-4945-93a7-7e100bcf37a9\") " Nov 01 22:46:16.957882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.957745 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22946ea8-86f3-4945-93a7-7e100bcf37a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22946ea8-86f3-4945-93a7-7e100bcf37a9-plugins-config-volume\") pod \"22946ea8-86f3-4945-93a7-7e100bcf37a9\" (UID: \"22946ea8-86f3-4945-93a7-7e100bcf37a9\") " Nov 01 22:46:16.958156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.958133 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22946ea8-86f3-4945-93a7-7e100bcf37a9-plugins-config-volume" pod "22946ea8-86f3-4945-93a7-7e100bcf37a9" (UID: "22946ea8-86f3-4945-93a7-7e100bcf37a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:16.958420 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.958401 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22946ea8-86f3-4945-93a7-7e100bcf37a9-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:16.959896 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.959873 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22946ea8-86f3-4945-93a7-7e100bcf37a9-kube-api-access-mc7m8" pod "22946ea8-86f3-4945-93a7-7e100bcf37a9" (UID: "22946ea8-86f3-4945-93a7-7e100bcf37a9"). InnerVolumeSpecName "kube-api-access-mc7m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:16.982047 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.982016 6466 generic.go:356] "Generic (PLEG): container finished" podID="22946ea8-86f3-4945-93a7-7e100bcf37a9" containerID="2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5" exitCode=0 Nov 01 22:46:16.982179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.982062 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" Nov 01 22:46:16.982179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.982069 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" event={"ID":"22946ea8-86f3-4945-93a7-7e100bcf37a9","Type":"ContainerDied","Data":"2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5"} Nov 01 22:46:16.982179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.982141 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz" event={"ID":"22946ea8-86f3-4945-93a7-7e100bcf37a9","Type":"ContainerDied","Data":"382ff7d7ec4510243a6a6bd859354586da208d515ca3bcda6127c9b57a978223"} Nov 01 22:46:16.982179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.982171 6466 scope.go:122] "RemoveContainer" containerID="2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5" Nov 01 22:46:16.994240 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.994222 6466 scope.go:122] "RemoveContainer" containerID="2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5" Nov 01 22:46:16.994499 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:16.994481 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5\": container with ID starting with 2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5 not found: ID does not exist" containerID="2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5" Nov 01 22:46:16.994568 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:16.994506 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5"} err="failed to get container status \"2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5\": rpc error: code = NotFound desc = could not find container \"2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5\": container with ID starting with 2e0e42c274eb516485d2eb05f1d2397d678acd59f91289147e3a68a4acb80ce5 not found: ID does not exist" Nov 01 22:46:17.012800 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:17.012775 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz"] Nov 01 22:46:17.016455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:17.016434 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ae0e3a-757674ccb7-z9rlz"] Nov 01 22:46:17.059975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:17.059897 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc7m8\" (UniqueName: \"kubernetes.io/projected/22946ea8-86f3-4945-93a7-7e100bcf37a9-kube-api-access-mc7m8\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:18.259775 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.259733 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" podUID="b6f22ca4-6c61-433b-9994-46af11ec66e4" containerName="payload-processing" containerID="cri-o://a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce" gracePeriod=25 Nov 01 22:46:18.270393 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.270358 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" podUID="dd9fe434-9161-4eb4-9aa2-795f3d28a633" containerName="payload-pre-processing" containerID="cri-o://360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1" gracePeriod=25 Nov 01 22:46:18.343371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.343337 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22946ea8-86f3-4945-93a7-7e100bcf37a9" path="/var/lib/kubelet/pods/22946ea8-86f3-4945-93a7-7e100bcf37a9/volumes" Nov 01 22:46:18.567530 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.567499 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:46:18.600975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.600943 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:46:18.695217 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.695181 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9fe434-9161-4eb4-9aa2-795f3d28a633-kube-api-access-hr4sw\" (UniqueName: \"kubernetes.io/projected/dd9fe434-9161-4eb4-9aa2-795f3d28a633-kube-api-access-hr4sw\") pod \"dd9fe434-9161-4eb4-9aa2-795f3d28a633\" (UID: \"dd9fe434-9161-4eb4-9aa2-795f3d28a633\") " Nov 01 22:46:18.695419 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.695401 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f22ca4-6c61-433b-9994-46af11ec66e4-kube-api-access-h7c6b\" (UniqueName: \"kubernetes.io/projected/b6f22ca4-6c61-433b-9994-46af11ec66e4-kube-api-access-h7c6b\") pod \"b6f22ca4-6c61-433b-9994-46af11ec66e4\" (UID: \"b6f22ca4-6c61-433b-9994-46af11ec66e4\") " Nov 01 22:46:18.695522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.695506 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f22ca4-6c61-433b-9994-46af11ec66e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f22ca4-6c61-433b-9994-46af11ec66e4-plugins-config-volume\") pod \"b6f22ca4-6c61-433b-9994-46af11ec66e4\" (UID: \"b6f22ca4-6c61-433b-9994-46af11ec66e4\") " Nov 01 22:46:18.695566 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.695553 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9fe434-9161-4eb4-9aa2-795f3d28a633-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9fe434-9161-4eb4-9aa2-795f3d28a633-plugins-config-volume\") pod \"dd9fe434-9161-4eb4-9aa2-795f3d28a633\" (UID: \"dd9fe434-9161-4eb4-9aa2-795f3d28a633\") " Nov 01 22:46:18.695886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.695860 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f22ca4-6c61-433b-9994-46af11ec66e4-plugins-config-volume" pod "b6f22ca4-6c61-433b-9994-46af11ec66e4" (UID: "b6f22ca4-6c61-433b-9994-46af11ec66e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:18.696001 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.695932 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9fe434-9161-4eb4-9aa2-795f3d28a633-plugins-config-volume" pod "dd9fe434-9161-4eb4-9aa2-795f3d28a633" (UID: "dd9fe434-9161-4eb4-9aa2-795f3d28a633"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:18.696335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.696311 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f22ca4-6c61-433b-9994-46af11ec66e4-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:18.696418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.696338 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd9fe434-9161-4eb4-9aa2-795f3d28a633-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:18.697334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.697307 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9fe434-9161-4eb4-9aa2-795f3d28a633-kube-api-access-hr4sw" pod "dd9fe434-9161-4eb4-9aa2-795f3d28a633" (UID: "dd9fe434-9161-4eb4-9aa2-795f3d28a633"). InnerVolumeSpecName "kube-api-access-hr4sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:18.697522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.697505 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f22ca4-6c61-433b-9994-46af11ec66e4-kube-api-access-h7c6b" pod "b6f22ca4-6c61-433b-9994-46af11ec66e4" (UID: "b6f22ca4-6c61-433b-9994-46af11ec66e4"). InnerVolumeSpecName "kube-api-access-h7c6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:18.800398 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.800318 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr4sw\" (UniqueName: \"kubernetes.io/projected/dd9fe434-9161-4eb4-9aa2-795f3d28a633-kube-api-access-hr4sw\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:18.800398 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.800355 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7c6b\" (UniqueName: \"kubernetes.io/projected/b6f22ca4-6c61-433b-9994-46af11ec66e4-kube-api-access-h7c6b\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:18.999978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.999943 6466 generic.go:356] "Generic (PLEG): container finished" podID="dd9fe434-9161-4eb4-9aa2-795f3d28a633" containerID="360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1" exitCode=0 Nov 01 22:46:19.000213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:18.999994 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" Nov 01 22:46:19.000213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.000032 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" event={"ID":"dd9fe434-9161-4eb4-9aa2-795f3d28a633","Type":"ContainerDied","Data":"360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1"} Nov 01 22:46:19.000213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.000090 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw" event={"ID":"dd9fe434-9161-4eb4-9aa2-795f3d28a633","Type":"ContainerDied","Data":"006a95ea802005cbfd4b3844b9d29ddccf0804aaefe9dfe0256652a556c107ff"} Nov 01 22:46:19.000213 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.000152 6466 scope.go:122] "RemoveContainer" containerID="360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1" Nov 01 22:46:19.001526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.001506 6466 generic.go:356] "Generic (PLEG): container finished" podID="b6f22ca4-6c61-433b-9994-46af11ec66e4" containerID="a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce" exitCode=0 Nov 01 22:46:19.001622 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.001555 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" Nov 01 22:46:19.001685 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.001551 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" event={"ID":"b6f22ca4-6c61-433b-9994-46af11ec66e4","Type":"ContainerDied","Data":"a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce"} Nov 01 22:46:19.001685 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.001676 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc" event={"ID":"b6f22ca4-6c61-433b-9994-46af11ec66e4","Type":"ContainerDied","Data":"d2fd2bc2432f22c901691e3acbd65359ae29e42dfd177e5f4ed563be65645ede"} Nov 01 22:46:19.012402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.012382 6466 scope.go:122] "RemoveContainer" containerID="360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1" Nov 01 22:46:19.012727 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:19.012703 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1\": container with ID starting with 360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1 not found: ID does not exist" containerID="360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1" Nov 01 22:46:19.012820 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.012733 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1"} err="failed to get container status \"360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1\": rpc error: code = NotFound desc = could not find container \"360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1\": container with ID starting with 360af41a16adf9ef19a592e0fd39990d572baf573bd1f1fe3d54973b16c216d1 not found: ID does not exist" Nov 01 22:46:19.012820 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.012749 6466 scope.go:122] "RemoveContainer" containerID="a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce" Nov 01 22:46:19.022940 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.022922 6466 scope.go:122] "RemoveContainer" containerID="a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce" Nov 01 22:46:19.023740 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:19.023275 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce\": container with ID starting with a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce not found: ID does not exist" containerID="a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce" Nov 01 22:46:19.023740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.023315 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce"} err="failed to get container status \"a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce\": rpc error: code = NotFound desc = could not find container \"a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce\": container with ID starting with a8ddf0dceb57e8808d79621f803d8e5640d9c5ec2d10c60e17a26cd7ec2305ce not found: ID does not exist" Nov 01 22:46:19.027347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.027327 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw"] Nov 01 22:46:19.033842 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.033821 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fa8cda-5bd7c8ff7f-tvjhw"] Nov 01 22:46:19.044002 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.043980 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc"] Nov 01 22:46:19.049907 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:19.049884 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fa8cda-99c8b9bf-58bgc"] Nov 01 22:46:19.283568 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:19.283539 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:46:20.414844 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:20.414806 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f22ca4-6c61-433b-9994-46af11ec66e4" path="/var/lib/kubelet/pods/b6f22ca4-6c61-433b-9994-46af11ec66e4/volumes" Nov 01 22:46:20.415390 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:20.415374 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9fe434-9161-4eb4-9aa2-795f3d28a633" path="/var/lib/kubelet/pods/dd9fe434-9161-4eb4-9aa2-795f3d28a633/volumes" Nov 01 22:46:26.402818 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.402778 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9"] Nov 01 22:46:26.405438 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405412 6466 cpu_manager.go:414] "containerMap: removing container" podUID="22946ea8-86f3-4945-93a7-7e100bcf37a9" containerName="payload-pre-processing" Nov 01 22:46:26.405438 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405434 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22946ea8-86f3-4945-93a7-7e100bcf37a9" containerName="payload-pre-processing" Nov 01 22:46:26.405438 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405447 6466 cpu_manager.go:414] "containerMap: removing container" podUID="dd9fe434-9161-4eb4-9aa2-795f3d28a633" containerName="payload-pre-processing" Nov 01 22:46:26.405438 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405452 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9fe434-9161-4eb4-9aa2-795f3d28a633" containerName="payload-pre-processing" Nov 01 22:46:26.405771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405460 6466 cpu_manager.go:414] "containerMap: removing container" podUID="b6f22ca4-6c61-433b-9994-46af11ec66e4" containerName="payload-processing" Nov 01 22:46:26.405771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405465 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f22ca4-6c61-433b-9994-46af11ec66e4" containerName="payload-processing" Nov 01 22:46:26.405771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405482 6466 cpu_manager.go:414] "containerMap: removing container" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" Nov 01 22:46:26.405771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405486 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" Nov 01 22:46:26.405771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405499 6466 cpu_manager.go:414] "containerMap: removing container" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" Nov 01 22:46:26.405771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405504 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" Nov 01 22:46:26.405771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405615 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" Nov 01 22:46:26.405771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405631 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="22946ea8-86f3-4945-93a7-7e100bcf37a9" containerName="payload-pre-processing" Nov 01 22:46:26.405771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405638 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9fe434-9161-4eb4-9aa2-795f3d28a633" containerName="payload-pre-processing" Nov 01 22:46:26.405771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.405645 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f22ca4-6c61-433b-9994-46af11ec66e4" containerName="payload-processing" Nov 01 22:46:26.411908 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.411887 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.419205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.419169 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9"] Nov 01 22:46:26.424075 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.424046 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1fddbd\"/\"kube-root-ca.crt\"" Nov 01 22:46:26.424250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.424160 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1fddbd\"/\"openshift-service-ca.crt\"" Nov 01 22:46:26.424250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.424242 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1fddbd\"/\"default-dockercfg-qcnth\"" Nov 01 22:46:26.424359 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.424321 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1fddbd\"/\"test-model-1fddbd-kserve-self-signed-certs\"" Nov 01 22:46:26.517580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.517547 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-model-cache\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.517734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.517587 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q242\" (UniqueName: \"kubernetes.io/projected/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-kube-api-access-7q242\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.517734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.517671 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-home\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.517814 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.517797 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-dshm\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.517860 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.517826 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tmp-dir\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.518015 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.518002 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tls-certs\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.626128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.621012 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-model-cache\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.626128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.621060 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q242\" (UniqueName: \"kubernetes.io/projected/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-kube-api-access-7q242\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.626128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.621149 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-home\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.626128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.621290 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-dshm\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.626128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.621329 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tmp-dir\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.626128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.621469 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tls-certs\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.626128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.623889 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-model-cache\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.642132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.639473 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-home\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.642132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.639754 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tmp-dir\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.672653 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.672501 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q242\" (UniqueName: \"kubernetes.io/projected/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-kube-api-access-7q242\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.711553 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.679490 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-dshm\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.711553 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.680181 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tls-certs\") pod \"test-model-1fddbd-kserve-597df9bcbf-zlrb9\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:26.735137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:26.733343 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:27.110814 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:27.110787 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9"] Nov 01 22:46:27.111005 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:46:27.110982 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4c02c43_5be8_41a0_a6e8_08ef1ee65472.slice/crio-157e17419eb0c2f5e2cd9d5cf2c9fac5ffaafdf6b54d676aa8117587651fb7ee WatchSource:0}: Error finding container 157e17419eb0c2f5e2cd9d5cf2c9fac5ffaafdf6b54d676aa8117587651fb7ee: Status 404 returned error can't find the container with id 157e17419eb0c2f5e2cd9d5cf2c9fac5ffaafdf6b54d676aa8117587651fb7ee Nov 01 22:46:27.177786 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:27.177757 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" event={"ID":"d4c02c43-5be8-41a0-a6e8-08ef1ee65472","Type":"ContainerStarted","Data":"157e17419eb0c2f5e2cd9d5cf2c9fac5ffaafdf6b54d676aa8117587651fb7ee"} Nov 01 22:46:28.186711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:28.186673 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" event={"ID":"d4c02c43-5be8-41a0-a6e8-08ef1ee65472","Type":"ContainerStarted","Data":"d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2"} Nov 01 22:46:28.212749 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:28.212692 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" podStartSLOduration=2.212679184 podStartE2EDuration="2.212679184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:28.209257693 +0000 UTC m=+1391.993696625" watchObservedRunningTime="2026-11-01 22:46:28.212679184 +0000 UTC m=+1391.997118097" Nov 01 22:46:28.734912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:28.734878 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:30.596152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:30.591322 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" containerID="cri-o://0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9" gracePeriod=45 Nov 01 22:46:30.602971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:30.602322 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" containerID="cri-o://9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00" gracePeriod=45 Nov 01 22:46:30.732301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:30.732263 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused" Nov 01 22:46:30.917298 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:30.917265 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:46:30.935123 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:30.935085 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:46:31.055247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055211 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-dshm\") pod \"5c7c32d2-89e8-4703-9775-0a7a168b318e\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " Nov 01 22:46:31.055591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055292 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c7c32d2-89e8-4703-9775-0a7a168b318e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7c32d2-89e8-4703-9775-0a7a168b318e-tls-certs\") pod \"5c7c32d2-89e8-4703-9775-0a7a168b318e\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " Nov 01 22:46:31.055591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055332 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-home\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-home\") pod \"5c7c32d2-89e8-4703-9775-0a7a168b318e\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " Nov 01 22:46:31.055591 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055372 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-model-cache\") pod \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " Nov 01 22:46:31.055808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055600 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-model-cache" pod "9ed3436f-6092-45a6-b2df-5d59b28d43bf" (UID: "9ed3436f-6092-45a6-b2df-5d59b28d43bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:31.055808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055628 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-home" pod "5c7c32d2-89e8-4703-9775-0a7a168b318e" (UID: "5c7c32d2-89e8-4703-9775-0a7a168b318e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:31.055808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055654 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-model-cache\") pod \"5c7c32d2-89e8-4703-9775-0a7a168b318e\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " Nov 01 22:46:31.055808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055724 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tls-certs\") pod \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " Nov 01 22:46:31.055808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055771 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-tmp-dir\") pod \"5c7c32d2-89e8-4703-9775-0a7a168b318e\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " Nov 01 22:46:31.056062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055827 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tmp-dir\") pod \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " Nov 01 22:46:31.056062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055893 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-dshm\") pod \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " Nov 01 22:46:31.056062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055940 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-model-cache" pod "5c7c32d2-89e8-4703-9775-0a7a168b318e" (UID: "5c7c32d2-89e8-4703-9775-0a7a168b318e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:31.056062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.055948 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-home\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-home\") pod \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " Nov 01 22:46:31.056272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.056187 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-home" pod "9ed3436f-6092-45a6-b2df-5d59b28d43bf" (UID: "9ed3436f-6092-45a6-b2df-5d59b28d43bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:31.056272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.056198 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7c32d2-89e8-4703-9775-0a7a168b318e-kube-api-access-65v9c\" (UniqueName: \"kubernetes.io/projected/5c7c32d2-89e8-4703-9775-0a7a168b318e-kube-api-access-65v9c\") pod \"5c7c32d2-89e8-4703-9775-0a7a168b318e\" (UID: \"5c7c32d2-89e8-4703-9775-0a7a168b318e\") " Nov 01 22:46:31.056368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.056341 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed3436f-6092-45a6-b2df-5d59b28d43bf-kube-api-access-5c8j4\" (UniqueName: \"kubernetes.io/projected/9ed3436f-6092-45a6-b2df-5d59b28d43bf-kube-api-access-5c8j4\") pod \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\" (UID: \"9ed3436f-6092-45a6-b2df-5d59b28d43bf\") " Nov 01 22:46:31.056417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.056379 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-tmp-dir" pod "5c7c32d2-89e8-4703-9775-0a7a168b318e" (UID: "5c7c32d2-89e8-4703-9775-0a7a168b318e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:31.056417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.056388 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tmp-dir" pod "9ed3436f-6092-45a6-b2df-5d59b28d43bf" (UID: "9ed3436f-6092-45a6-b2df-5d59b28d43bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:31.057159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.057127 6466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-home\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.057271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.057162 6466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-home\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.057271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.057176 6466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-model-cache\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.057271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.057189 6466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-model-cache\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.057271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.057201 6466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-tmp-dir\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.057271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.057213 6466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tmp-dir\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.058356 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.058324 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c7c32d2-89e8-4703-9775-0a7a168b318e-tls-certs" pod "5c7c32d2-89e8-4703-9775-0a7a168b318e" (UID: "5c7c32d2-89e8-4703-9775-0a7a168b318e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:31.059609 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.059584 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7c32d2-89e8-4703-9775-0a7a168b318e-kube-api-access-65v9c" pod "5c7c32d2-89e8-4703-9775-0a7a168b318e" (UID: "5c7c32d2-89e8-4703-9775-0a7a168b318e"). InnerVolumeSpecName "kube-api-access-65v9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:31.059971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.059947 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-dshm" pod "9ed3436f-6092-45a6-b2df-5d59b28d43bf" (UID: "9ed3436f-6092-45a6-b2df-5d59b28d43bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:31.060279 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.060248 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tls-certs" pod "9ed3436f-6092-45a6-b2df-5d59b28d43bf" (UID: "9ed3436f-6092-45a6-b2df-5d59b28d43bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:31.060364 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.060280 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed3436f-6092-45a6-b2df-5d59b28d43bf-kube-api-access-5c8j4" pod "9ed3436f-6092-45a6-b2df-5d59b28d43bf" (UID: "9ed3436f-6092-45a6-b2df-5d59b28d43bf"). InnerVolumeSpecName "kube-api-access-5c8j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:31.061010 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.060979 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-dshm" pod "5c7c32d2-89e8-4703-9775-0a7a168b318e" (UID: "5c7c32d2-89e8-4703-9775-0a7a168b318e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:31.159920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.159789 6466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c7c32d2-89e8-4703-9775-0a7a168b318e-dshm\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.159920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.159829 6466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c7c32d2-89e8-4703-9775-0a7a168b318e-tls-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.159920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.159844 6466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed3436f-6092-45a6-b2df-5d59b28d43bf-tls-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.159920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.159857 6466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed3436f-6092-45a6-b2df-5d59b28d43bf-dshm\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.159920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.159871 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65v9c\" (UniqueName: \"kubernetes.io/projected/5c7c32d2-89e8-4703-9775-0a7a168b318e-kube-api-access-65v9c\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.159920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.159887 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c8j4\" (UniqueName: \"kubernetes.io/projected/9ed3436f-6092-45a6-b2df-5d59b28d43bf-kube-api-access-5c8j4\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.213099 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.213065 6466 generic.go:356] "Generic (PLEG): container finished" podID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerID="9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00" exitCode=0 Nov 01 22:46:31.213312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.213185 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" Nov 01 22:46:31.213312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.213176 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" event={"ID":"5c7c32d2-89e8-4703-9775-0a7a168b318e","Type":"ContainerDied","Data":"9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00"} Nov 01 22:46:31.213424 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.213334 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs" event={"ID":"5c7c32d2-89e8-4703-9775-0a7a168b318e","Type":"ContainerDied","Data":"96572498bb62084fe89c6e8d58e4376552a7e6f91da9c44554ebb612d84c66ce"} Nov 01 22:46:31.213424 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.213356 6466 scope.go:122] "RemoveContainer" containerID="9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00" Nov 01 22:46:31.214889 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.214867 6466 generic.go:356] "Generic (PLEG): container finished" podID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerID="0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9" exitCode=0 Nov 01 22:46:31.214979 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.214946 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" Nov 01 22:46:31.214979 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.214958 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" event={"ID":"9ed3436f-6092-45a6-b2df-5d59b28d43bf","Type":"ContainerDied","Data":"0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9"} Nov 01 22:46:31.215093 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.214985 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx" event={"ID":"9ed3436f-6092-45a6-b2df-5d59b28d43bf","Type":"ContainerDied","Data":"2afb237680ca83922cdf992638a0623ac6e7a224df3240340ffe5eef54a9b48a"} Nov 01 22:46:31.226721 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.226699 6466 scope.go:122] "RemoveContainer" containerID="9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00" Nov 01 22:46:31.227016 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:31.226993 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00\": container with ID starting with 9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00 not found: ID does not exist" containerID="9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00" Nov 01 22:46:31.227250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.227026 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00"} err="failed to get container status \"9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00\": rpc error: code = NotFound desc = could not find container \"9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00\": container with ID starting with 9e0d91153b7e138872a6d8ae5ada66a190a81a61adcb69ccde2d32fda9f17d00 not found: ID does not exist" Nov 01 22:46:31.227250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.227049 6466 scope.go:122] "RemoveContainer" containerID="0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9" Nov 01 22:46:31.240420 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.240397 6466 scope.go:122] "RemoveContainer" containerID="0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9" Nov 01 22:46:31.240743 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:31.240723 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9\": container with ID starting with 0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9 not found: ID does not exist" containerID="0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9" Nov 01 22:46:31.240822 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.240753 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9"} err="failed to get container status \"0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9\": rpc error: code = NotFound desc = could not find container \"0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9\": container with ID starting with 0b4250dfa0a67b6bf4417da10b504061dcb26d5a5e50174e1ccc94347386d5c9 not found: ID does not exist" Nov 01 22:46:31.256741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.256707 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs"] Nov 01 22:46:31.260759 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.260728 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-w3-00e49d-kserve-6485b75c54-h8lxs"] Nov 01 22:46:31.281881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.281847 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx"] Nov 01 22:46:31.288825 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:31.288796 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-00e49d/distinct-2-w3-00e49d-kserve-86974ff7fc-h27fx"] Nov 01 22:46:32.343971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:32.343918 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" path="/var/lib/kubelet/pods/5c7c32d2-89e8-4703-9775-0a7a168b318e/volumes" Nov 01 22:46:32.346576 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:32.344520 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" path="/var/lib/kubelet/pods/9ed3436f-6092-45a6-b2df-5d59b28d43bf/volumes" Nov 01 22:46:33.234503 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:33.234471 6466 generic.go:356] "Generic (PLEG): container finished" podID="a07b617c-2d6b-4814-af51-406639a9b920" containerID="13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23" exitCode=0 Nov 01 22:46:33.234823 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:33.234547 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" event={"ID":"a07b617c-2d6b-4814-af51-406639a9b920","Type":"ContainerDied","Data":"13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23"} Nov 01 22:46:33.235098 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:33.235081 6466 scope.go:122] "RemoveContainer" containerID="13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23" Nov 01 22:46:33.237047 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:33.237013 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:46:33.237159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:33.237103 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:46:33.237203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:33.237166 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:46:33.546196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:33.537224 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:46:34.280766 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:34.280733 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" event={"ID":"a07b617c-2d6b-4814-af51-406639a9b920","Type":"ContainerStarted","Data":"824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735"} Nov 01 22:46:34.287338 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:34.287307 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq"] Nov 01 22:46:34.287783 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:34.287732 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" podUID="402f118e-e4ef-4bfe-9e0f-ed5003167964" containerName="maas-api" containerID="cri-o://05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21" gracePeriod=30 Nov 01 22:46:34.660964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:34.646455 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:46:34.660964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:34.647853 6466 patch_prober.go:29] interesting pod/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 22:46:34.660964 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:34.647905 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 22:46:35.113130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.111925 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:46:35.194392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.194298 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-metrics-certs\" (UniqueName: \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-metrics-certs\") pod \"402f118e-e4ef-4bfe-9e0f-ed5003167964\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " Nov 01 22:46:35.194771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.194750 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-maas-api-tls\") pod \"402f118e-e4ef-4bfe-9e0f-ed5003167964\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " Nov 01 22:46:35.194919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.194903 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402f118e-e4ef-4bfe-9e0f-ed5003167964-kube-api-access-hhcn4\" (UniqueName: \"kubernetes.io/projected/402f118e-e4ef-4bfe-9e0f-ed5003167964-kube-api-access-hhcn4\") pod \"402f118e-e4ef-4bfe-9e0f-ed5003167964\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " Nov 01 22:46:35.195236 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.195215 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402f118e-e4ef-4bfe-9e0f-ed5003167964-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f118e-e4ef-4bfe-9e0f-ed5003167964-openshift-service-ca\") pod \"402f118e-e4ef-4bfe-9e0f-ed5003167964\" (UID: \"402f118e-e4ef-4bfe-9e0f-ed5003167964\") " Nov 01 22:46:35.211405 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.211365 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402f118e-e4ef-4bfe-9e0f-ed5003167964-openshift-service-ca" pod "402f118e-e4ef-4bfe-9e0f-ed5003167964" (UID: "402f118e-e4ef-4bfe-9e0f-ed5003167964"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:35.215605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.215555 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-metrics-certs" pod "402f118e-e4ef-4bfe-9e0f-ed5003167964" (UID: "402f118e-e4ef-4bfe-9e0f-ed5003167964"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:35.216627 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.216216 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-maas-api-tls" pod "402f118e-e4ef-4bfe-9e0f-ed5003167964" (UID: "402f118e-e4ef-4bfe-9e0f-ed5003167964"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:35.222051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.220798 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402f118e-e4ef-4bfe-9e0f-ed5003167964-kube-api-access-hhcn4" pod "402f118e-e4ef-4bfe-9e0f-ed5003167964" (UID: "402f118e-e4ef-4bfe-9e0f-ed5003167964"). InnerVolumeSpecName "kube-api-access-hhcn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:35.290720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.290688 6466 generic.go:356] "Generic (PLEG): container finished" podID="402f118e-e4ef-4bfe-9e0f-ed5003167964" containerID="05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21" exitCode=0 Nov 01 22:46:35.290930 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.290752 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" Nov 01 22:46:35.290930 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.290825 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" event={"ID":"402f118e-e4ef-4bfe-9e0f-ed5003167964","Type":"ContainerDied","Data":"05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21"} Nov 01 22:46:35.290930 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.290863 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq" event={"ID":"402f118e-e4ef-4bfe-9e0f-ed5003167964","Type":"ContainerDied","Data":"b246e7a2313df663c082cd5a57f6ff947d2058804c242a5f70b89755633e1e1e"} Nov 01 22:46:35.290930 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.290884 6466 scope.go:122] "RemoveContainer" containerID="05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21" Nov 01 22:46:35.302866 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.302001 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:35.302866 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.302033 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhcn4\" (UniqueName: \"kubernetes.io/projected/402f118e-e4ef-4bfe-9e0f-ed5003167964-kube-api-access-hhcn4\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:35.302866 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.302055 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/402f118e-e4ef-4bfe-9e0f-ed5003167964-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:35.302866 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.302071 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/402f118e-e4ef-4bfe-9e0f-ed5003167964-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:35.311459 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.311433 6466 scope.go:122] "RemoveContainer" containerID="05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21" Nov 01 22:46:35.311900 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:35.311865 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21\": container with ID starting with 05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21 not found: ID does not exist" containerID="05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21" Nov 01 22:46:35.312031 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.312012 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21"} err="failed to get container status \"05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21\": rpc error: code = NotFound desc = could not find container \"05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21\": container with ID starting with 05fd66db43917559311565b054a53704444452a1655497f7f9a41896ce2e7b21 not found: ID does not exist" Nov 01 22:46:35.374409 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.372202 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq"] Nov 01 22:46:35.378561 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.378532 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ae0e3a-cf586ccc5-lq4mq"] Nov 01 22:46:35.599188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.598319 6466 patch_prober.go:29] interesting pod/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 22:46:35.599188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:35.598396 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 22:46:36.367764 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:36.367732 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402f118e-e4ef-4bfe-9e0f-ed5003167964" path="/var/lib/kubelet/pods/402f118e-e4ef-4bfe-9e0f-ed5003167964/volumes" Nov 01 22:46:36.585206 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:36.585173 6466 patch_prober.go:29] interesting pod/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 22:46:36.585408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:36.585226 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 22:46:36.752143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:36.751343 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:36.771998 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:36.771972 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:37.318511 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.318483 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:46:37.369401 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.369364 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287"] Nov 01 22:46:37.370652 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.370618 6466 cpu_manager.go:414] "containerMap: removing container" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" Nov 01 22:46:37.370652 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.370650 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" Nov 01 22:46:37.370852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.370677 6466 cpu_manager.go:414] "containerMap: removing container" podUID="402f118e-e4ef-4bfe-9e0f-ed5003167964" containerName="maas-api" Nov 01 22:46:37.370852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.370684 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402f118e-e4ef-4bfe-9e0f-ed5003167964" containerName="maas-api" Nov 01 22:46:37.370852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.370711 6466 cpu_manager.go:414] "containerMap: removing container" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" Nov 01 22:46:37.370852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.370718 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" Nov 01 22:46:37.371039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.370932 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8ca67a-a7ed-4a7c-905e-f02763779799" containerName="istio-proxy" Nov 01 22:46:37.371039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.370952 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7c32d2-89e8-4703-9775-0a7a168b318e" containerName="main" Nov 01 22:46:37.371039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.370964 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="402f118e-e4ef-4bfe-9e0f-ed5003167964" containerName="maas-api" Nov 01 22:46:37.371039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.370975 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed3436f-6092-45a6-b2df-5d59b28d43bf" containerName="main" Nov 01 22:46:37.380221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.377400 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.386281 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.386255 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9717e2\"/\"default-dockercfg-7mvl8\"" Nov 01 22:46:37.389976 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.389952 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287"] Nov 01 22:46:37.391919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.391892 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9717e2\"/\"test-model-9717e2-kserve-self-signed-certs\"" Nov 01 22:46:37.392634 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.392612 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9717e2\"/\"kube-root-ca.crt\"" Nov 01 22:46:37.392723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.392698 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9717e2\"/\"openshift-service-ca.crt\"" Nov 01 22:46:37.469517 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.469480 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2nwg\" (UniqueName: \"kubernetes.io/projected/eb5806d0-ea66-468f-bc19-e38198c6e135-kube-api-access-v2nwg\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.469689 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.469538 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-tmp-dir\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.469750 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.469699 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-home\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.469807 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.469751 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-model-cache\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.469882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.469856 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5806d0-ea66-468f-bc19-e38198c6e135-tls-certs\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.469942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.469920 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-dshm\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.596183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.592522 6466 patch_prober.go:29] interesting pod/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 22:46:37.596183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.592587 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 22:46:37.596183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.594127 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2nwg\" (UniqueName: \"kubernetes.io/projected/eb5806d0-ea66-468f-bc19-e38198c6e135-kube-api-access-v2nwg\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.596183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.594194 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-tmp-dir\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.596183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.594326 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-home\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.596183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.594384 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-model-cache\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.596183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.594414 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5806d0-ea66-468f-bc19-e38198c6e135-tls-certs\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.596183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.594459 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-dshm\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.596183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.595211 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-home\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.596183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.595818 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-tmp-dir\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.596183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.596082 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-model-cache\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.643701 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.643661 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2nwg\" (UniqueName: \"kubernetes.io/projected/eb5806d0-ea66-468f-bc19-e38198c6e135-kube-api-access-v2nwg\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.659141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.651175 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-dshm\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.664502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.664435 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5806d0-ea66-468f-bc19-e38198c6e135-tls-certs\") pod \"test-model-9717e2-kserve-75bf6f7645-mb287\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:37.698572 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:37.698313 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:38.103632 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:38.103541 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2"] Nov 01 22:46:38.104201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:38.104169 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" containerName="istio-proxy" containerID="cri-o://ed55ee8cc11f45ca456d5a6891d7bdc8551095232587c3878a24bedbc7046b36" gracePeriod=30 Nov 01 22:46:38.121045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:38.121020 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287"] Nov 01 22:46:38.125843 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:46:38.125761 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb5806d0_ea66_468f_bc19_e38198c6e135.slice/crio-bcdd91ceff7a46cef6854956cb8e4ac5263380684def37492fba68a564fb0537 WatchSource:0}: Error finding container bcdd91ceff7a46cef6854956cb8e4ac5263380684def37492fba68a564fb0537: Status 404 returned error can't find the container with id bcdd91ceff7a46cef6854956cb8e4ac5263380684def37492fba68a564fb0537 Nov 01 22:46:38.316961 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:38.316919 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" event={"ID":"eb5806d0-ea66-468f-bc19-e38198c6e135","Type":"ContainerStarted","Data":"a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a"} Nov 01 22:46:38.317133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:38.316978 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" event={"ID":"eb5806d0-ea66-468f-bc19-e38198c6e135","Type":"ContainerStarted","Data":"bcdd91ceff7a46cef6854956cb8e4ac5263380684def37492fba68a564fb0537"} Nov 01 22:46:38.361600 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:38.361529 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" podStartSLOduration=1.361509865 podStartE2EDuration="1.361509865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:38.354445373 +0000 UTC m=+1402.138884299" watchObservedRunningTime="2026-11-01 22:46:38.361509865 +0000 UTC m=+1402.145948787" Nov 01 22:46:38.584753 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:38.584717 6466 patch_prober.go:29] interesting pod/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 22:46:38.585194 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:38.584772 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 22:46:38.699791 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:38.699754 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:39.584813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:39.584776 6466 patch_prober.go:29] interesting pod/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 22:46:39.585226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:39.584845 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 22:46:40.589672 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:40.589642 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:46:40.590064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:40.590045 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:46:40.590616 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:40.590595 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:46:43.377066 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.377033 6466 generic.go:356] "Generic (PLEG): container finished" podID="16da46c6-bb20-4107-b575-a886d44abd6e" containerID="ed55ee8cc11f45ca456d5a6891d7bdc8551095232587c3878a24bedbc7046b36" exitCode=0 Nov 01 22:46:43.377480 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.377215 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" event={"ID":"16da46c6-bb20-4107-b575-a886d44abd6e","Type":"ContainerDied","Data":"ed55ee8cc11f45ca456d5a6891d7bdc8551095232587c3878a24bedbc7046b36"} Nov 01 22:46:43.556505 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.556474 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv"] Nov 01 22:46:43.556914 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.556883 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" containerID="cri-o://9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f" gracePeriod=30 Nov 01 22:46:43.563864 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.563835 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:46:43.592901 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.592874 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-socket\") pod \"16da46c6-bb20-4107-b575-a886d44abd6e\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " Nov 01 22:46:43.593024 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.592922 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-certs\") pod \"16da46c6-bb20-4107-b575-a886d44abd6e\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " Nov 01 22:46:43.593024 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.592951 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-credential-socket\") pod \"16da46c6-bb20-4107-b575-a886d44abd6e\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " Nov 01 22:46:43.593024 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.593016 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-data\") pod \"16da46c6-bb20-4107-b575-a886d44abd6e\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " Nov 01 22:46:43.593225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.593096 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-istio-token\" (UniqueName: \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-istio-token\") pod \"16da46c6-bb20-4107-b575-a886d44abd6e\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " Nov 01 22:46:43.593225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.593183 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-socket" pod "16da46c6-bb20-4107-b575-a886d44abd6e" (UID: "16da46c6-bb20-4107-b575-a886d44abd6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:43.593225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.593209 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-kube-api-access-v9xp8\" (UniqueName: \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-kube-api-access-v9xp8\") pod \"16da46c6-bb20-4107-b575-a886d44abd6e\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " Nov 01 22:46:43.593402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.593224 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-certs" pod "16da46c6-bb20-4107-b575-a886d44abd6e" (UID: "16da46c6-bb20-4107-b575-a886d44abd6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:43.593402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.593257 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16da46c6-bb20-4107-b575-a886d44abd6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16da46c6-bb20-4107-b575-a886d44abd6e-istio-podinfo\") pod \"16da46c6-bb20-4107-b575-a886d44abd6e\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " Nov 01 22:46:43.593402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.593304 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16da46c6-bb20-4107-b575-a886d44abd6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16da46c6-bb20-4107-b575-a886d44abd6e-istiod-ca-cert\") pod \"16da46c6-bb20-4107-b575-a886d44abd6e\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " Nov 01 22:46:43.593402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.593317 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-credential-socket" pod "16da46c6-bb20-4107-b575-a886d44abd6e" (UID: "16da46c6-bb20-4107-b575-a886d44abd6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:43.593402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.593401 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-envoy\") pod \"16da46c6-bb20-4107-b575-a886d44abd6e\" (UID: \"16da46c6-bb20-4107-b575-a886d44abd6e\") " Nov 01 22:46:43.593673 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.593625 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16da46c6-bb20-4107-b575-a886d44abd6e-istiod-ca-cert" pod "16da46c6-bb20-4107-b575-a886d44abd6e" (UID: "16da46c6-bb20-4107-b575-a886d44abd6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:43.594164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.594089 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-data" pod "16da46c6-bb20-4107-b575-a886d44abd6e" (UID: "16da46c6-bb20-4107-b575-a886d44abd6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:43.594288 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.594255 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16da46c6-bb20-4107-b575-a886d44abd6e-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:43.594288 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.594282 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:43.594402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.594296 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:43.594402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.594310 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:43.595489 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.595459 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-istio-token" pod "16da46c6-bb20-4107-b575-a886d44abd6e" (UID: "16da46c6-bb20-4107-b575-a886d44abd6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:43.595578 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.595485 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-kube-api-access-v9xp8" pod "16da46c6-bb20-4107-b575-a886d44abd6e" (UID: "16da46c6-bb20-4107-b575-a886d44abd6e"). InnerVolumeSpecName "kube-api-access-v9xp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:43.595772 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.595755 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16da46c6-bb20-4107-b575-a886d44abd6e-istio-podinfo" pod "16da46c6-bb20-4107-b575-a886d44abd6e" (UID: "16da46c6-bb20-4107-b575-a886d44abd6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:43.595814 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.595797 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-envoy" pod "16da46c6-bb20-4107-b575-a886d44abd6e" (UID: "16da46c6-bb20-4107-b575-a886d44abd6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:43.696190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.696152 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:43.696190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.696181 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:43.696190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.696195 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9xp8\" (UniqueName: \"kubernetes.io/projected/16da46c6-bb20-4107-b575-a886d44abd6e-kube-api-access-v9xp8\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:43.696401 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.696208 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16da46c6-bb20-4107-b575-a886d44abd6e-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:43.696401 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:43.696222 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16da46c6-bb20-4107-b575-a886d44abd6e-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:44.384870 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:44.384842 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" Nov 01 22:46:44.385371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:44.384878 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2" event={"ID":"16da46c6-bb20-4107-b575-a886d44abd6e","Type":"ContainerDied","Data":"6daefffbd353ce9d3295f35d8e4da413a12a24debd8195df24bd6541558dd4bb"} Nov 01 22:46:44.385371 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:44.384945 6466 scope.go:122] "RemoveContainer" containerID="ed55ee8cc11f45ca456d5a6891d7bdc8551095232587c3878a24bedbc7046b36" Nov 01 22:46:44.410432 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:44.410411 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2"] Nov 01 22:46:44.414453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:44.414428 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-htjp2"] Nov 01 22:46:45.705322 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:45.705289 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88"] Nov 01 22:46:45.707229 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:45.707196 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" podUID="c704a06d-d1e6-4e05-8730-15cb9f7a4309" containerName="maas-api" containerID="cri-o://4bd5f7d166c929b5699ae8b98a475ac56df9fd4fb26db76d1ee1f743f995a949" gracePeriod=30 Nov 01 22:46:46.409094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.404351 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" path="/var/lib/kubelet/pods/16da46c6-bb20-4107-b575-a886d44abd6e/volumes" Nov 01 22:46:46.409094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.406329 6466 generic.go:356] "Generic (PLEG): container finished" podID="c704a06d-d1e6-4e05-8730-15cb9f7a4309" containerID="4bd5f7d166c929b5699ae8b98a475ac56df9fd4fb26db76d1ee1f743f995a949" exitCode=0 Nov 01 22:46:46.409094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.406439 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" event={"ID":"c704a06d-d1e6-4e05-8730-15cb9f7a4309","Type":"ContainerDied","Data":"4bd5f7d166c929b5699ae8b98a475ac56df9fd4fb26db76d1ee1f743f995a949"} Nov 01 22:46:46.409094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.406469 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" event={"ID":"c704a06d-d1e6-4e05-8730-15cb9f7a4309","Type":"ContainerDied","Data":"ba8ae3d9b1b19dacf07d38dbf703c823b0468ebc86a83958098e6e01ac115d60"} Nov 01 22:46:46.409094 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.406484 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba8ae3d9b1b19dacf07d38dbf703c823b0468ebc86a83958098e6e01ac115d60" Nov 01 22:46:46.420162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.418287 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:46:46.599895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.599807 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-maas-api-tls\") pod \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " Nov 01 22:46:46.600087 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.600034 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c704a06d-d1e6-4e05-8730-15cb9f7a4309-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c704a06d-d1e6-4e05-8730-15cb9f7a4309-openshift-service-ca\") pod \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " Nov 01 22:46:46.600188 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.600136 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-metrics-certs\") pod \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " Nov 01 22:46:46.600255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.600201 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c704a06d-d1e6-4e05-8730-15cb9f7a4309-kube-api-access-pj4tf\" (UniqueName: \"kubernetes.io/projected/c704a06d-d1e6-4e05-8730-15cb9f7a4309-kube-api-access-pj4tf\") pod \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\" (UID: \"c704a06d-d1e6-4e05-8730-15cb9f7a4309\") " Nov 01 22:46:46.600647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.600618 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c704a06d-d1e6-4e05-8730-15cb9f7a4309-openshift-service-ca" pod "c704a06d-d1e6-4e05-8730-15cb9f7a4309" (UID: "c704a06d-d1e6-4e05-8730-15cb9f7a4309"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:46.601211 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.601135 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c704a06d-d1e6-4e05-8730-15cb9f7a4309-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:46.603167 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.603085 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-maas-api-tls" pod "c704a06d-d1e6-4e05-8730-15cb9f7a4309" (UID: "c704a06d-d1e6-4e05-8730-15cb9f7a4309"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:46.614750 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.614713 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-metrics-certs" pod "c704a06d-d1e6-4e05-8730-15cb9f7a4309" (UID: "c704a06d-d1e6-4e05-8730-15cb9f7a4309"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:46.614750 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.614721 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c704a06d-d1e6-4e05-8730-15cb9f7a4309-kube-api-access-pj4tf" pod "c704a06d-d1e6-4e05-8730-15cb9f7a4309" (UID: "c704a06d-d1e6-4e05-8730-15cb9f7a4309"). InnerVolumeSpecName "kube-api-access-pj4tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:46.702838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.702805 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:46.702838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.702832 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c704a06d-d1e6-4e05-8730-15cb9f7a4309-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:46.702838 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:46.702843 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4tf\" (UniqueName: \"kubernetes.io/projected/c704a06d-d1e6-4e05-8730-15cb9f7a4309-kube-api-access-pj4tf\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:47.191368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:47.191333 6466 patch_prober.go:29] interesting pod/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:46:47.191764 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:47.191417 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:46:47.414258 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:47.414231 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88" Nov 01 22:46:47.459054 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:47.458977 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88"] Nov 01 22:46:47.475681 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:47.475655 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-00e49d-597594b94-rqg88"] Nov 01 22:46:47.699095 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:47.699063 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:47.711644 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:47.711573 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:48.342968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.342937 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c704a06d-d1e6-4e05-8730-15cb9f7a4309" path="/var/lib/kubelet/pods/c704a06d-d1e6-4e05-8730-15cb9f7a4309/volumes" Nov 01 22:46:48.470135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.469331 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:46:48.837980 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.837954 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:46:48.862808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.862782 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-envoy\") pod \"43636efd-d17f-4e43-bb3d-f82b63999337\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " Nov 01 22:46:48.862961 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.862877 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-istio-token\" (UniqueName: \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-istio-token\") pod \"43636efd-d17f-4e43-bb3d-f82b63999337\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " Nov 01 22:46:48.862961 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.862948 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43636efd-d17f-4e43-bb3d-f82b63999337-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43636efd-d17f-4e43-bb3d-f82b63999337-istio-podinfo\") pod \"43636efd-d17f-4e43-bb3d-f82b63999337\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " Nov 01 22:46:48.863078 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.863023 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43636efd-d17f-4e43-bb3d-f82b63999337-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43636efd-d17f-4e43-bb3d-f82b63999337-istiod-ca-cert\") pod \"43636efd-d17f-4e43-bb3d-f82b63999337\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " Nov 01 22:46:48.863185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.863168 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-kube-api-access-llccn\" (UniqueName: \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-kube-api-access-llccn\") pod \"43636efd-d17f-4e43-bb3d-f82b63999337\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " Nov 01 22:46:48.863259 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.863235 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-data\") pod \"43636efd-d17f-4e43-bb3d-f82b63999337\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " Nov 01 22:46:48.863359 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.863345 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-certs\") pod \"43636efd-d17f-4e43-bb3d-f82b63999337\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " Nov 01 22:46:48.863423 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.863396 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-credential-socket\") pod \"43636efd-d17f-4e43-bb3d-f82b63999337\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " Nov 01 22:46:48.863480 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.863461 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-socket\") pod \"43636efd-d17f-4e43-bb3d-f82b63999337\" (UID: \"43636efd-d17f-4e43-bb3d-f82b63999337\") " Nov 01 22:46:48.863847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.863710 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-certs" pod "43636efd-d17f-4e43-bb3d-f82b63999337" (UID: "43636efd-d17f-4e43-bb3d-f82b63999337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:48.863847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.863776 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-data" pod "43636efd-d17f-4e43-bb3d-f82b63999337" (UID: "43636efd-d17f-4e43-bb3d-f82b63999337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:48.863847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.863794 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-credential-socket" pod "43636efd-d17f-4e43-bb3d-f82b63999337" (UID: "43636efd-d17f-4e43-bb3d-f82b63999337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:48.864051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.863987 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-socket" pod "43636efd-d17f-4e43-bb3d-f82b63999337" (UID: "43636efd-d17f-4e43-bb3d-f82b63999337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:48.864171 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.864069 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43636efd-d17f-4e43-bb3d-f82b63999337-istiod-ca-cert" pod "43636efd-d17f-4e43-bb3d-f82b63999337" (UID: "43636efd-d17f-4e43-bb3d-f82b63999337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:48.864392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.864372 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:48.864427 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.864399 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:48.864427 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.864415 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43636efd-d17f-4e43-bb3d-f82b63999337-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:48.864491 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.864429 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:48.864491 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.864443 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:48.865268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.865246 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-envoy" pod "43636efd-d17f-4e43-bb3d-f82b63999337" (UID: "43636efd-d17f-4e43-bb3d-f82b63999337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:48.865268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.865258 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-istio-token" pod "43636efd-d17f-4e43-bb3d-f82b63999337" (UID: "43636efd-d17f-4e43-bb3d-f82b63999337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:48.865541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.865521 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43636efd-d17f-4e43-bb3d-f82b63999337-istio-podinfo" pod "43636efd-d17f-4e43-bb3d-f82b63999337" (UID: "43636efd-d17f-4e43-bb3d-f82b63999337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:48.865604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.865561 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-kube-api-access-llccn" pod "43636efd-d17f-4e43-bb3d-f82b63999337" (UID: "43636efd-d17f-4e43-bb3d-f82b63999337"). InnerVolumeSpecName "kube-api-access-llccn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:48.966141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.966041 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llccn\" (UniqueName: \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-kube-api-access-llccn\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:48.966141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.966074 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43636efd-d17f-4e43-bb3d-f82b63999337-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:48.966141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.966085 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43636efd-d17f-4e43-bb3d-f82b63999337-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:48.966141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:48.966094 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43636efd-d17f-4e43-bb3d-f82b63999337-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:49.037204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.037168 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k"] Nov 01 22:46:49.283495 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:49.283405 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:46:49.460533 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.460497 6466 generic.go:356] "Generic (PLEG): container finished" podID="43636efd-d17f-4e43-bb3d-f82b63999337" containerID="9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f" exitCode=0 Nov 01 22:46:49.460919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.460876 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" event={"ID":"43636efd-d17f-4e43-bb3d-f82b63999337","Type":"ContainerDied","Data":"9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f"} Nov 01 22:46:49.460984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.460937 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" event={"ID":"43636efd-d17f-4e43-bb3d-f82b63999337","Type":"ContainerDied","Data":"d56aae6232c2009991fccd4b10aa76fbe1f1662a430766444c4fcc8e92667616"} Nov 01 22:46:49.460984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.460969 6466 scope.go:122] "RemoveContainer" containerID="9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f" Nov 01 22:46:49.461064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.461051 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv" Nov 01 22:46:49.472304 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.472286 6466 scope.go:122] "RemoveContainer" containerID="7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c" Nov 01 22:46:49.482350 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.482332 6466 scope.go:122] "RemoveContainer" containerID="9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f" Nov 01 22:46:49.482614 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:49.482596 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f\": container with ID starting with 9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f not found: ID does not exist" containerID="9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f" Nov 01 22:46:49.482656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.482621 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f"} err="failed to get container status \"9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f\": rpc error: code = NotFound desc = could not find container \"9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f\": container with ID starting with 9ad62df40c2857fc2468711c0c62c89858be4d27196add9b844e6f18f56ba65f not found: ID does not exist" Nov 01 22:46:49.482656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.482639 6466 scope.go:122] "RemoveContainer" containerID="7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c" Nov 01 22:46:49.482895 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:46:49.482877 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c\": container with ID starting with 7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c not found: ID does not exist" containerID="7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c" Nov 01 22:46:49.482950 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.482899 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c"} err="failed to get container status \"7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c\": rpc error: code = NotFound desc = could not find container \"7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c\": container with ID starting with 7e6b2739571cf67cd947430027b1467ebdb44bb23fb7aef39c0c8e5aa6d2341c not found: ID does not exist" Nov 01 22:46:49.499439 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.499417 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv"] Nov 01 22:46:49.509485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:49.509463 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ae0e3a-openshift-default-5f4964f445-j77rv"] Nov 01 22:46:50.351015 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:50.350975 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" path="/var/lib/kubelet/pods/43636efd-d17f-4e43-bb3d-f82b63999337/volumes" Nov 01 22:46:50.778183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:50.778152 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z"] Nov 01 22:46:50.778978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:50.778924 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" containerID="cri-o://e8db45fdaf8a5ebe4aa15781d4630f8521596de384e9b1bac70388560788f059" gracePeriod=30 Nov 01 22:46:54.436839 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:54.436800 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t"] Nov 01 22:46:56.569669 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.569636 6466 generic.go:356] "Generic (PLEG): container finished" podID="101175a2-a477-494a-99ff-ac688c53c07e" containerID="e8db45fdaf8a5ebe4aa15781d4630f8521596de384e9b1bac70388560788f059" exitCode=0 Nov 01 22:46:56.570075 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.569828 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" event={"ID":"101175a2-a477-494a-99ff-ac688c53c07e","Type":"ContainerDied","Data":"e8db45fdaf8a5ebe4aa15781d4630f8521596de384e9b1bac70388560788f059"} Nov 01 22:46:56.570075 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.570056 6466 scope.go:122] "RemoveContainer" containerID="b01b22ec0d0d3a6a351b46ad2b60a5f7a70ee174fe1cbe3fcd146eca65e4e4d6" Nov 01 22:46:56.593031 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.593010 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:46:56.653145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653094 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-socket\") pod \"101175a2-a477-494a-99ff-ac688c53c07e\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " Nov 01 22:46:56.653307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653203 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-credential-socket\") pod \"101175a2-a477-494a-99ff-ac688c53c07e\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " Nov 01 22:46:56.653395 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653303 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101175a2-a477-494a-99ff-ac688c53c07e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101175a2-a477-494a-99ff-ac688c53c07e-istiod-ca-cert\") pod \"101175a2-a477-494a-99ff-ac688c53c07e\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " Nov 01 22:46:56.653395 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653339 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-certs\") pod \"101175a2-a477-494a-99ff-ac688c53c07e\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " Nov 01 22:46:56.653516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653393 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-istio-token\" (UniqueName: \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-istio-token\") pod \"101175a2-a477-494a-99ff-ac688c53c07e\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " Nov 01 22:46:56.653516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653478 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-socket" pod "101175a2-a477-494a-99ff-ac688c53c07e" (UID: "101175a2-a477-494a-99ff-ac688c53c07e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:56.653516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653505 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-kube-api-access-rzt95\" (UniqueName: \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-kube-api-access-rzt95\") pod \"101175a2-a477-494a-99ff-ac688c53c07e\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " Nov 01 22:46:56.653686 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653548 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-envoy\") pod \"101175a2-a477-494a-99ff-ac688c53c07e\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " Nov 01 22:46:56.653686 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653560 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-credential-socket" pod "101175a2-a477-494a-99ff-ac688c53c07e" (UID: "101175a2-a477-494a-99ff-ac688c53c07e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:56.653686 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653630 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/101175a2-a477-494a-99ff-ac688c53c07e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101175a2-a477-494a-99ff-ac688c53c07e-istio-podinfo\") pod \"101175a2-a477-494a-99ff-ac688c53c07e\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " Nov 01 22:46:56.653863 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653680 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-certs" pod "101175a2-a477-494a-99ff-ac688c53c07e" (UID: "101175a2-a477-494a-99ff-ac688c53c07e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:56.653863 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.653691 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-data\") pod \"101175a2-a477-494a-99ff-ac688c53c07e\" (UID: \"101175a2-a477-494a-99ff-ac688c53c07e\") " Nov 01 22:46:56.654942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.654286 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101175a2-a477-494a-99ff-ac688c53c07e-istiod-ca-cert" pod "101175a2-a477-494a-99ff-ac688c53c07e" (UID: "101175a2-a477-494a-99ff-ac688c53c07e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:56.654942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.654464 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-data" pod "101175a2-a477-494a-99ff-ac688c53c07e" (UID: "101175a2-a477-494a-99ff-ac688c53c07e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:56.654942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.654656 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.654942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.654679 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101175a2-a477-494a-99ff-ac688c53c07e-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.654942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.654695 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.654942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.654708 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.654942 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.654722 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.656073 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.656042 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-istio-token" pod "101175a2-a477-494a-99ff-ac688c53c07e" (UID: "101175a2-a477-494a-99ff-ac688c53c07e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:56.656225 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.656136 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-kube-api-access-rzt95" pod "101175a2-a477-494a-99ff-ac688c53c07e" (UID: "101175a2-a477-494a-99ff-ac688c53c07e"). InnerVolumeSpecName "kube-api-access-rzt95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:56.656517 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.656490 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/101175a2-a477-494a-99ff-ac688c53c07e-istio-podinfo" pod "101175a2-a477-494a-99ff-ac688c53c07e" (UID: "101175a2-a477-494a-99ff-ac688c53c07e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:56.656754 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.656726 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-envoy" pod "101175a2-a477-494a-99ff-ac688c53c07e" (UID: "101175a2-a477-494a-99ff-ac688c53c07e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:56.756492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.756315 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzt95\" (UniqueName: \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-kube-api-access-rzt95\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.756492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.756342 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101175a2-a477-494a-99ff-ac688c53c07e-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.756492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.756351 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101175a2-a477-494a-99ff-ac688c53c07e-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.756492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:56.756359 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101175a2-a477-494a-99ff-ac688c53c07e-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:57.583374 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:57.583328 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" event={"ID":"101175a2-a477-494a-99ff-ac688c53c07e","Type":"ContainerDied","Data":"1a50b73399d1d7bbe59f34c3b91dd74ddcee038763b601281f5aa3692d65da7d"} Nov 01 22:46:57.583930 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:57.583450 6466 scope.go:122] "RemoveContainer" containerID="e8db45fdaf8a5ebe4aa15781d4630f8521596de384e9b1bac70388560788f059" Nov 01 22:46:57.583930 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:57.583729 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z" Nov 01 22:46:57.629603 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:57.629573 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z"] Nov 01 22:46:57.637262 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:57.637236 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-c9a7ce-openshift-default-867b799c87-45j5z"] Nov 01 22:46:58.343722 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:58.343686 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101175a2-a477-494a-99ff-ac688c53c07e" path="/var/lib/kubelet/pods/101175a2-a477-494a-99ff-ac688c53c07e/volumes" Nov 01 22:46:58.425129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:58.422247 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2"] Nov 01 22:46:58.425129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:58.422472 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" containerID="cri-o://1bee0ba21cb23c9694c8aa7605f2dac71635252144b5ab5872ab011e4a5c6739" gracePeriod=30 Nov 01 22:46:58.429179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:58.427523 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph"] Nov 01 22:46:58.429179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:58.427735 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" podUID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" containerName="istio-proxy" containerID="cri-o://f5f3dfd65610a547b18be189d9ec00596b10b66d104d5d810ed74db07acfa667" gracePeriod=30 Nov 01 22:46:59.437383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:46:59.437314 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" podUID="2e5fd4c8-80e4-4169-a6a7-30cbee6941b8" containerName="payload-pre-processing" containerID="cri-o://5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e" gracePeriod=25 Nov 01 22:47:01.331511 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:01.331478 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt"] Nov 01 22:47:03.665541 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.662230 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" podUID="2e5fd4c8-80e4-4169-a6a7-30cbee6941b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:47:03.670648 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.670606 6466 generic.go:356] "Generic (PLEG): container finished" podID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerID="1bee0ba21cb23c9694c8aa7605f2dac71635252144b5ab5872ab011e4a5c6739" exitCode=0 Nov 01 22:47:03.671152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.670824 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" event={"ID":"ec6ba517-374b-45a0-b36c-43d08dc7fffa","Type":"ContainerDied","Data":"1bee0ba21cb23c9694c8aa7605f2dac71635252144b5ab5872ab011e4a5c6739"} Nov 01 22:47:03.682610 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.682547 6466 generic.go:356] "Generic (PLEG): container finished" podID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" containerID="f5f3dfd65610a547b18be189d9ec00596b10b66d104d5d810ed74db07acfa667" exitCode=0 Nov 01 22:47:03.683051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.683024 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" event={"ID":"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375","Type":"ContainerDied","Data":"f5f3dfd65610a547b18be189d9ec00596b10b66d104d5d810ed74db07acfa667"} Nov 01 22:47:03.846174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.846144 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:47:03.850671 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.850645 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:47:03.963250 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963206 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-data\") pod \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " Nov 01 22:47:03.963469 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963266 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-credential-socket\") pod \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " Nov 01 22:47:03.963469 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963326 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-kube-api-access-fdgpr\" (UniqueName: \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-kube-api-access-fdgpr\") pod \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " Nov 01 22:47:03.963469 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963355 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-data\") pod \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " Nov 01 22:47:03.963469 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963410 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-envoy\") pod \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " Nov 01 22:47:03.963469 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963450 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-credential-socket\") pod \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " Nov 01 22:47:03.963746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963486 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-socket\") pod \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " Nov 01 22:47:03.963746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963537 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istiod-ca-cert\") pod \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " Nov 01 22:47:03.963746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963564 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-data" pod "ec6ba517-374b-45a0-b36c-43d08dc7fffa" (UID: "ec6ba517-374b-45a0-b36c-43d08dc7fffa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.963746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963568 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-credential-socket" pod "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" (UID: "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.963746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963572 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istiod-ca-cert\") pod \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " Nov 01 22:47:03.963746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963718 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-data" pod "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" (UID: "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.964059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963768 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-certs\") pod \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " Nov 01 22:47:03.964059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963816 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-token\" (UniqueName: \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-token\") pod \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " Nov 01 22:47:03.964059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963827 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-socket" pod "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" (UID: "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.964059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963857 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-podinfo\") pod \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " Nov 01 22:47:03.964059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963896 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-kube-api-access-kdqdz\" (UniqueName: \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-kube-api-access-kdqdz\") pod \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " Nov 01 22:47:03.964059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963945 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-socket\") pod \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " Nov 01 22:47:03.964059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.963966 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istiod-ca-cert" pod "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" (UID: "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:03.964458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.964063 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istiod-ca-cert" pod "ec6ba517-374b-45a0-b36c-43d08dc7fffa" (UID: "ec6ba517-374b-45a0-b36c-43d08dc7fffa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:03.964458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.964065 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-certs" pod "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" (UID: "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.964458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.964191 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-credential-socket" pod "ec6ba517-374b-45a0-b36c-43d08dc7fffa" (UID: "ec6ba517-374b-45a0-b36c-43d08dc7fffa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.964458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.964228 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-socket" pod "ec6ba517-374b-45a0-b36c-43d08dc7fffa" (UID: "ec6ba517-374b-45a0-b36c-43d08dc7fffa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.964458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.964230 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-certs\") pod \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " Nov 01 22:47:03.964458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.964285 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-podinfo\") pod \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\" (UID: \"ec6ba517-374b-45a0-b36c-43d08dc7fffa\") " Nov 01 22:47:03.964458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.964332 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-token\" (UniqueName: \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-token\") pod \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " Nov 01 22:47:03.964458 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.964406 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-envoy\") pod \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\" (UID: \"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375\") " Nov 01 22:47:03.964916 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.964652 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-certs" pod "ec6ba517-374b-45a0-b36c-43d08dc7fffa" (UID: "ec6ba517-374b-45a0-b36c-43d08dc7fffa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.965206 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.965179 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.965319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.965211 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.965319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.965226 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.965319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.965240 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.965319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.965253 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.965319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.965266 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.965319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.965280 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.965319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.965294 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.965319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.965307 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.965319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.965321 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.966592 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.966563 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-podinfo" pod "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" (UID: "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:03.966724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.966593 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-token" pod "ec6ba517-374b-45a0-b36c-43d08dc7fffa" (UID: "ec6ba517-374b-45a0-b36c-43d08dc7fffa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:03.966724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.966613 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-kube-api-access-fdgpr" pod "ec6ba517-374b-45a0-b36c-43d08dc7fffa" (UID: "ec6ba517-374b-45a0-b36c-43d08dc7fffa"). InnerVolumeSpecName "kube-api-access-fdgpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:03.966895 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.966872 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-kube-api-access-kdqdz" pod "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" (UID: "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375"). InnerVolumeSpecName "kube-api-access-kdqdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:03.967057 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.967033 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-envoy" pod "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" (UID: "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.967280 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.967260 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-envoy" pod "ec6ba517-374b-45a0-b36c-43d08dc7fffa" (UID: "ec6ba517-374b-45a0-b36c-43d08dc7fffa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.967448 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.967424 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-token" pod "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" (UID: "c4da916a-f9ac-4cd2-ad7d-4c6dd034f375"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:03.967545 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:03.967501 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-podinfo" pod "ec6ba517-374b-45a0-b36c-43d08dc7fffa" (UID: "ec6ba517-374b-45a0-b36c-43d08dc7fffa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:04.059162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.059037 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" containerID="cri-o://5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c" gracePeriod=45 Nov 01 22:47:04.066247 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.066224 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.066347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.066248 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.066347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.066260 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdgpr\" (UniqueName: \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-kube-api-access-fdgpr\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.066347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.066270 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.066347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.066279 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.066347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.066287 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.066347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.066295 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdqdz\" (UniqueName: \"kubernetes.io/projected/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375-kube-api-access-kdqdz\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.066347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.066304 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6ba517-374b-45a0-b36c-43d08dc7fffa-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.327472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.327441 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:47:04.480608 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.480576 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-model-cache\") pod \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " Nov 01 22:47:04.480777 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.480717 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tls-certs\") pod \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " Nov 01 22:47:04.480828 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.480817 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tmp-dir\") pod \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " Nov 01 22:47:04.480886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.480870 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc77f6d-f23f-4191-bc0e-ba1abc195039-kube-api-access-8lhfs\" (UniqueName: \"kubernetes.io/projected/2bc77f6d-f23f-4191-bc0e-ba1abc195039-kube-api-access-8lhfs\") pod \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " Nov 01 22:47:04.480984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.480958 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-model-cache" pod "2bc77f6d-f23f-4191-bc0e-ba1abc195039" (UID: "2bc77f6d-f23f-4191-bc0e-ba1abc195039"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.481039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.481001 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-home\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-home\") pod \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " Nov 01 22:47:04.481277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.481255 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-dshm\") pod \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\" (UID: \"2bc77f6d-f23f-4191-bc0e-ba1abc195039\") " Nov 01 22:47:04.481407 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.481279 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-home" pod "2bc77f6d-f23f-4191-bc0e-ba1abc195039" (UID: "2bc77f6d-f23f-4191-bc0e-ba1abc195039"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.482152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.482100 6466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-home\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.482268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.482154 6466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-model-cache\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.482409 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.482387 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tmp-dir" pod "2bc77f6d-f23f-4191-bc0e-ba1abc195039" (UID: "2bc77f6d-f23f-4191-bc0e-ba1abc195039"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.485244 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.485221 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-dshm" pod "2bc77f6d-f23f-4191-bc0e-ba1abc195039" (UID: "2bc77f6d-f23f-4191-bc0e-ba1abc195039"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.486634 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.486608 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tls-certs" pod "2bc77f6d-f23f-4191-bc0e-ba1abc195039" (UID: "2bc77f6d-f23f-4191-bc0e-ba1abc195039"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:04.486806 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.486781 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc77f6d-f23f-4191-bc0e-ba1abc195039-kube-api-access-8lhfs" pod "2bc77f6d-f23f-4191-bc0e-ba1abc195039" (UID: "2bc77f6d-f23f-4191-bc0e-ba1abc195039"). InnerVolumeSpecName "kube-api-access-8lhfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:04.583948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.583859 6466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-dshm\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.583948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.583892 6466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tls-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.583948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.583905 6466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc77f6d-f23f-4191-bc0e-ba1abc195039-tmp-dir\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.583948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.583920 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lhfs\" (UniqueName: \"kubernetes.io/projected/2bc77f6d-f23f-4191-bc0e-ba1abc195039-kube-api-access-8lhfs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.691204 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.691175 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" Nov 01 22:47:04.691678 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.691178 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2" event={"ID":"ec6ba517-374b-45a0-b36c-43d08dc7fffa","Type":"ContainerDied","Data":"c90a972343118bafd692145f3e94a4f0b67ae36ab548c79151d9375cc7660a92"} Nov 01 22:47:04.691678 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.691303 6466 scope.go:122] "RemoveContainer" containerID="1bee0ba21cb23c9694c8aa7605f2dac71635252144b5ab5872ab011e4a5c6739" Nov 01 22:47:04.693045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.693022 6466 generic.go:356] "Generic (PLEG): container finished" podID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerID="5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c" exitCode=0 Nov 01 22:47:04.693209 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.693076 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" Nov 01 22:47:04.693209 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.693103 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" event={"ID":"2bc77f6d-f23f-4191-bc0e-ba1abc195039","Type":"ContainerDied","Data":"5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c"} Nov 01 22:47:04.693322 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.693223 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k" event={"ID":"2bc77f6d-f23f-4191-bc0e-ba1abc195039","Type":"ContainerDied","Data":"79d4b2504513ba778eed731a6b416c140ddb1ef012da7dd6734750831cfea163"} Nov 01 22:47:04.694907 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.694884 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" event={"ID":"c4da916a-f9ac-4cd2-ad7d-4c6dd034f375","Type":"ContainerDied","Data":"2fe4dfe4166b19fdac8928a57669921c07d801c5176606f17199cffd87981877"} Nov 01 22:47:04.695033 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.694936 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph" Nov 01 22:47:04.704453 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.704433 6466 scope.go:122] "RemoveContainer" containerID="5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c" Nov 01 22:47:04.713185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.713160 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2"] Nov 01 22:47:04.714558 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.714543 6466 scope.go:122] "RemoveContainer" containerID="5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c" Nov 01 22:47:04.714816 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:04.714799 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c\": container with ID starting with 5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c not found: ID does not exist" containerID="5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c" Nov 01 22:47:04.714861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.714823 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c"} err="failed to get container status \"5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c\": rpc error: code = NotFound desc = could not find container \"5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c\": container with ID starting with 5a228ef3e0d8da438fd871f98f459d75c3cbcfa6a8bba3da4d0c0b2c26169b3c not found: ID does not exist" Nov 01 22:47:04.714861 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.714839 6466 scope.go:122] "RemoveContainer" containerID="f5f3dfd65610a547b18be189d9ec00596b10b66d104d5d810ed74db07acfa667" Nov 01 22:47:04.720772 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.720745 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zndd2"] Nov 01 22:47:04.742508 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.742476 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph"] Nov 01 22:47:04.747853 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.747828 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-6zpph"] Nov 01 22:47:04.768047 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.768014 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k"] Nov 01 22:47:04.778546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:04.778520 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-381659/distinct-2-w1-381659-kserve-6d7c4d78cc-rqp5k"] Nov 01 22:47:06.332232 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.332189 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" podUID="3092930b-ddb0-4915-8967-34ef095ae412" containerName="payload-processing" containerID="cri-o://d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8" gracePeriod=25 Nov 01 22:47:06.342678 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.342654 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" path="/var/lib/kubelet/pods/2bc77f6d-f23f-4191-bc0e-ba1abc195039/volumes" Nov 01 22:47:06.343179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.343163 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" path="/var/lib/kubelet/pods/c4da916a-f9ac-4cd2-ad7d-4c6dd034f375/volumes" Nov 01 22:47:06.343637 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.343624 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" path="/var/lib/kubelet/pods/ec6ba517-374b-45a0-b36c-43d08dc7fffa/volumes" Nov 01 22:47:06.615952 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.615927 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:47:06.714301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.714263 6466 generic.go:356] "Generic (PLEG): container finished" podID="3092930b-ddb0-4915-8967-34ef095ae412" containerID="d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8" exitCode=0 Nov 01 22:47:06.714496 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.714310 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" Nov 01 22:47:06.714496 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.714351 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" event={"ID":"3092930b-ddb0-4915-8967-34ef095ae412","Type":"ContainerDied","Data":"d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8"} Nov 01 22:47:06.714496 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.714435 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt" event={"ID":"3092930b-ddb0-4915-8967-34ef095ae412","Type":"ContainerDied","Data":"5f381d647c6a9c86f3f4c8a5cb81e4af10a0f77bcb1cb2d1fd89b8d3bfe62c8e"} Nov 01 22:47:06.714496 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.714461 6466 scope.go:122] "RemoveContainer" containerID="d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8" Nov 01 22:47:06.724869 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.724843 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3092930b-ddb0-4915-8967-34ef095ae412-kube-api-access-ktqfr\" (UniqueName: \"kubernetes.io/projected/3092930b-ddb0-4915-8967-34ef095ae412-kube-api-access-ktqfr\") pod \"3092930b-ddb0-4915-8967-34ef095ae412\" (UID: \"3092930b-ddb0-4915-8967-34ef095ae412\") " Nov 01 22:47:06.725071 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.725051 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3092930b-ddb0-4915-8967-34ef095ae412-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3092930b-ddb0-4915-8967-34ef095ae412-plugins-config-volume\") pod \"3092930b-ddb0-4915-8967-34ef095ae412\" (UID: \"3092930b-ddb0-4915-8967-34ef095ae412\") " Nov 01 22:47:06.725398 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.725376 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3092930b-ddb0-4915-8967-34ef095ae412-plugins-config-volume" pod "3092930b-ddb0-4915-8967-34ef095ae412" (UID: "3092930b-ddb0-4915-8967-34ef095ae412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:06.725752 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.725730 6466 scope.go:122] "RemoveContainer" containerID="d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8" Nov 01 22:47:06.725847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.725777 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3092930b-ddb0-4915-8967-34ef095ae412-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:06.726072 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:06.726053 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8\": container with ID starting with d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8 not found: ID does not exist" containerID="d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8" Nov 01 22:47:06.726182 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.726081 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8"} err="failed to get container status \"d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8\": rpc error: code = NotFound desc = could not find container \"d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8\": container with ID starting with d21048a3fbbbf56e5ecdcb2d8f6153545a0fe5fcf7e6799dfcf96d59989ac0c8 not found: ID does not exist" Nov 01 22:47:06.727500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.727475 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3092930b-ddb0-4915-8967-34ef095ae412-kube-api-access-ktqfr" pod "3092930b-ddb0-4915-8967-34ef095ae412" (UID: "3092930b-ddb0-4915-8967-34ef095ae412"). InnerVolumeSpecName "kube-api-access-ktqfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:06.827261 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:06.827226 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktqfr\" (UniqueName: \"kubernetes.io/projected/3092930b-ddb0-4915-8967-34ef095ae412-kube-api-access-ktqfr\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:07.039931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:07.039895 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt"] Nov 01 22:47:07.042766 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:07.042740 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3fa400-588cc4c98c-gmmnt"] Nov 01 22:47:08.341511 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:08.341477 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3092930b-ddb0-4915-8967-34ef095ae412" path="/var/lib/kubelet/pods/3092930b-ddb0-4915-8967-34ef095ae412/volumes" Nov 01 22:47:13.640959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:13.640919 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" podUID="2e5fd4c8-80e4-4169-a6a7-30cbee6941b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:47:14.741618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.741580 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd"] Nov 01 22:47:14.742599 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742575 6466 cpu_manager.go:414] "containerMap: removing container" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" Nov 01 22:47:14.742599 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742601 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" Nov 01 22:47:14.742784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742634 6466 cpu_manager.go:414] "containerMap: removing container" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" containerName="istio-proxy" Nov 01 22:47:14.742784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742644 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" containerName="istio-proxy" Nov 01 22:47:14.742784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742685 6466 cpu_manager.go:414] "containerMap: removing container" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" Nov 01 22:47:14.742784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742692 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" Nov 01 22:47:14.742784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742705 6466 cpu_manager.go:414] "containerMap: removing container" podUID="3092930b-ddb0-4915-8967-34ef095ae412" containerName="payload-processing" Nov 01 22:47:14.742784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742714 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3092930b-ddb0-4915-8967-34ef095ae412" containerName="payload-processing" Nov 01 22:47:14.742784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742724 6466 cpu_manager.go:414] "containerMap: removing container" podUID="c704a06d-d1e6-4e05-8730-15cb9f7a4309" containerName="maas-api" Nov 01 22:47:14.742784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742730 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c704a06d-d1e6-4e05-8730-15cb9f7a4309" containerName="maas-api" Nov 01 22:47:14.742784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742753 6466 cpu_manager.go:414] "containerMap: removing container" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" Nov 01 22:47:14.742784 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742760 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742794 6466 cpu_manager.go:414] "containerMap: removing container" podUID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" containerName="istio-proxy" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742802 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" containerName="istio-proxy" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742812 6466 cpu_manager.go:414] "containerMap: removing container" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742819 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742832 6466 cpu_manager.go:414] "containerMap: removing container" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.742839 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.743020 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.743033 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.743043 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="3092930b-ddb0-4915-8967-34ef095ae412" containerName="payload-processing" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.743056 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="c704a06d-d1e6-4e05-8730-15cb9f7a4309" containerName="maas-api" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.743068 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6ba517-374b-45a0-b36c-43d08dc7fffa" containerName="istio-proxy" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.743077 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4da916a-f9ac-4cd2-ad7d-4c6dd034f375" containerName="istio-proxy" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.743085 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="43636efd-d17f-4e43-bb3d-f82b63999337" containerName="istio-proxy" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.743094 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc77f6d-f23f-4191-bc0e-ba1abc195039" containerName="main" Nov 01 22:47:14.743302 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.743126 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="16da46c6-bb20-4107-b575-a886d44abd6e" containerName="istio-proxy" Nov 01 22:47:14.749899 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.749870 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.765667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.765636 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd"] Nov 01 22:47:14.855442 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.855402 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-credential-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.855656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.855469 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.855656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.855535 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-podinfo\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.855656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.855642 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-data\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.855845 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.855666 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-envoy\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.855845 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.855689 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-token\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.855845 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.855774 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istiod-ca-cert\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.856051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.856021 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbdd\" (UniqueName: \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-kube-api-access-5cbdd\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.856139 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.856069 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-certs\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.957516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.957481 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.957717 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.957551 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-podinfo\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.957717 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.957655 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-data\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.957717 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.957680 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-envoy\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.957717 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.957704 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-token\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.957957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.957788 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istiod-ca-cert\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.957957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.957874 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbdd\" (UniqueName: \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-kube-api-access-5cbdd\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.957957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.957903 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-certs\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.957957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.957944 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.958274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.958004 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-credential-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.958274 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.958008 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-data\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.958426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.958394 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-credential-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.958523 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.958505 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-certs\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.958950 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.958920 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istiod-ca-cert\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.960504 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.960480 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-envoy\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.960793 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.960770 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-podinfo\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.967552 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.967528 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbdd\" (UniqueName: \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-kube-api-access-5cbdd\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:14.971802 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:14.971778 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-token\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:15.069670 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:15.069589 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:15.412284 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:15.412254 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd"] Nov 01 22:47:15.414301 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:47:15.414256 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf681a6d9_5d1e_49f5_975d_7a0c56b20329.slice/crio-57b398a1748e6bb162268425f059e0f44cedc86240a0e9d86759d6cb4f292f4b WatchSource:0}: Error finding container 57b398a1748e6bb162268425f059e0f44cedc86240a0e9d86759d6cb4f292f4b: Status 404 returned error can't find the container with id 57b398a1748e6bb162268425f059e0f44cedc86240a0e9d86759d6cb4f292f4b Nov 01 22:47:15.417035 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:15.417004 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:47:15.417127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:15.417067 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:47:15.417127 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:15.417099 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 22:47:15.872387 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:15.872279 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" event={"ID":"f681a6d9-5d1e-49f5-975d-7a0c56b20329","Type":"ContainerStarted","Data":"35ee1cb9d174006c8f8154cb8f22473d8b2164302221670dfe2b4eaa5736d1b9"} Nov 01 22:47:15.873607 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:15.872413 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" event={"ID":"f681a6d9-5d1e-49f5-975d-7a0c56b20329","Type":"ContainerStarted","Data":"57b398a1748e6bb162268425f059e0f44cedc86240a0e9d86759d6cb4f292f4b"} Nov 01 22:47:15.902409 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:15.902350 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" podStartSLOduration=1.90233704 podStartE2EDuration="1.90233704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:15.901267151 +0000 UTC m=+1439.685706095" watchObservedRunningTime="2026-11-01 22:47:15.90233704 +0000 UTC m=+1439.686775955" Nov 01 22:47:16.070203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:16.070161 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:16.071736 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:16.071711 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:47:16.071859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:16.071759 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 22:47:17.070331 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:17.070295 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:47:17.070738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:17.070363 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 22:47:18.089133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:18.079531 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:47:18.089133 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:18.079591 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 22:47:19.076129 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.076038 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:47:19.076357 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.076130 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 22:47:19.381726 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:19.381627 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:47:19.784359 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.784326 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:47:19.902716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.902687 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-plugins-config-volume\") pod \"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8\" (UID: \"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8\") " Nov 01 22:47:19.902920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.902896 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-kube-api-access-tslgv\" (UniqueName: \"kubernetes.io/projected/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-kube-api-access-tslgv\") pod \"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8\" (UID: \"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8\") " Nov 01 22:47:19.903145 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.903098 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-plugins-config-volume" pod "2e5fd4c8-80e4-4169-a6a7-30cbee6941b8" (UID: "2e5fd4c8-80e4-4169-a6a7-30cbee6941b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:19.903418 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.903400 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:19.904999 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.904975 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-kube-api-access-tslgv" pod "2e5fd4c8-80e4-4169-a6a7-30cbee6941b8" (UID: "2e5fd4c8-80e4-4169-a6a7-30cbee6941b8"). InnerVolumeSpecName "kube-api-access-tslgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:19.924940 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.924882 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq"] Nov 01 22:47:19.996240 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.996205 6466 generic.go:356] "Generic (PLEG): container finished" podID="2e5fd4c8-80e4-4169-a6a7-30cbee6941b8" containerID="5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e" exitCode=0 Nov 01 22:47:19.996426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.996248 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" Nov 01 22:47:19.996426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.996282 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" event={"ID":"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8","Type":"ContainerDied","Data":"5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e"} Nov 01 22:47:19.996426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.996352 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t" event={"ID":"2e5fd4c8-80e4-4169-a6a7-30cbee6941b8","Type":"ContainerDied","Data":"ca6ac8e36e4ee9f55c7ee45bfe5a703dd1fa55fc3ae09bb1e6fe5a99ad94e110"} Nov 01 22:47:19.996426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:19.996374 6466 scope.go:122] "RemoveContainer" containerID="5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e" Nov 01 22:47:20.004542 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.004521 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tslgv\" (UniqueName: \"kubernetes.io/projected/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8-kube-api-access-tslgv\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:20.006824 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.006807 6466 scope.go:122] "RemoveContainer" containerID="5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e" Nov 01 22:47:20.007081 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:20.007060 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e\": container with ID starting with 5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e not found: ID does not exist" containerID="5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e" Nov 01 22:47:20.007174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.007092 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e"} err="failed to get container status \"5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e\": rpc error: code = NotFound desc = could not find container \"5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e\": container with ID starting with 5c3e8938a1d61de9def62430ff70b09cd753ee9cd49bd4cd2fbb3d574aa7a61e not found: ID does not exist" Nov 01 22:47:20.025886 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.025861 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t"] Nov 01 22:47:20.028597 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.028579 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-381659-587d544667-7r74t"] Nov 01 22:47:20.070848 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.070812 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:47:20.071014 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.070861 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 22:47:20.304923 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.303758 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn"] Nov 01 22:47:20.304923 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.303993 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" containerID="cri-o://741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4" gracePeriod=30 Nov 01 22:47:20.323037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.312722 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr"] Nov 01 22:47:20.323037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.312919 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" containerID="cri-o://151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263" gracePeriod=30 Nov 01 22:47:20.375045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:20.375010 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5fd4c8-80e4-4169-a6a7-30cbee6941b8" path="/var/lib/kubelet/pods/2e5fd4c8-80e4-4169-a6a7-30cbee6941b8/volumes" Nov 01 22:47:21.070648 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:21.070612 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:47:21.071061 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:21.070669 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 22:47:21.240943 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:21.234761 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm"] Nov 01 22:47:21.240943 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:21.235652 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" containerID="cri-o://281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b" gracePeriod=30 Nov 01 22:47:21.245383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:21.245352 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx"] Nov 01 22:47:21.245595 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:21.245573 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" containerID="cri-o://70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84" gracePeriod=30 Nov 01 22:47:21.690288 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:21.690256 6466 scope.go:122] "RemoveContainer" containerID="4bd5f7d166c929b5699ae8b98a475ac56df9fd4fb26db76d1ee1f743f995a949" Nov 01 22:47:22.070352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:22.070270 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:47:22.070352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:22.070322 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 22:47:23.074233 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:23.074197 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:23.074734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:23.074607 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:23.075382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:23.075362 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:47:25.104130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.095905 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4"] Nov 01 22:47:25.135630 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.135596 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl"] Nov 01 22:47:25.723442 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.723406 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:47:25.753368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.753347 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:47:25.818224 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818189 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-data\") pod \"f1daa908-9812-4ecf-923d-160fca8742c7\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " Nov 01 22:47:25.818397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818295 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-envoy\") pod \"53147cfc-27a6-4f61-863a-732bfd063831\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " Nov 01 22:47:25.818477 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818393 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-istio-token\" (UniqueName: \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-istio-token\") pod \"f1daa908-9812-4ecf-923d-160fca8742c7\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " Nov 01 22:47:25.818477 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818459 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-certs\") pod \"53147cfc-27a6-4f61-863a-732bfd063831\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " Nov 01 22:47:25.818584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818508 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-kube-api-access-hdtqt\" (UniqueName: \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-kube-api-access-hdtqt\") pod \"53147cfc-27a6-4f61-863a-732bfd063831\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " Nov 01 22:47:25.818584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818510 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-data" pod "f1daa908-9812-4ecf-923d-160fca8742c7" (UID: "f1daa908-9812-4ecf-923d-160fca8742c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.818584 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818554 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-data\") pod \"53147cfc-27a6-4f61-863a-732bfd063831\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " Nov 01 22:47:25.818743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818609 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-certs\") pod \"f1daa908-9812-4ecf-923d-160fca8742c7\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " Nov 01 22:47:25.818743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818660 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-credential-socket\") pod \"f1daa908-9812-4ecf-923d-160fca8742c7\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " Nov 01 22:47:25.818743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818694 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53147cfc-27a6-4f61-863a-732bfd063831-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53147cfc-27a6-4f61-863a-732bfd063831-istio-podinfo\") pod \"53147cfc-27a6-4f61-863a-732bfd063831\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " Nov 01 22:47:25.818743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818738 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1daa908-9812-4ecf-923d-160fca8742c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1daa908-9812-4ecf-923d-160fca8742c7-istio-podinfo\") pod \"f1daa908-9812-4ecf-923d-160fca8742c7\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " Nov 01 22:47:25.818960 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818784 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-socket\") pod \"f1daa908-9812-4ecf-923d-160fca8742c7\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " Nov 01 22:47:25.818960 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818824 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-socket\") pod \"53147cfc-27a6-4f61-863a-732bfd063831\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " Nov 01 22:47:25.818960 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818858 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1daa908-9812-4ecf-923d-160fca8742c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1daa908-9812-4ecf-923d-160fca8742c7-istiod-ca-cert\") pod \"f1daa908-9812-4ecf-923d-160fca8742c7\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " Nov 01 22:47:25.818960 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818889 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-credential-socket\") pod \"53147cfc-27a6-4f61-863a-732bfd063831\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " Nov 01 22:47:25.818960 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.818945 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-envoy\") pod \"f1daa908-9812-4ecf-923d-160fca8742c7\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " Nov 01 22:47:25.819335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.819015 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-certs" pod "53147cfc-27a6-4f61-863a-732bfd063831" (UID: "53147cfc-27a6-4f61-863a-732bfd063831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.819335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.819076 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53147cfc-27a6-4f61-863a-732bfd063831-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53147cfc-27a6-4f61-863a-732bfd063831-istiod-ca-cert\") pod \"53147cfc-27a6-4f61-863a-732bfd063831\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " Nov 01 22:47:25.819335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.819151 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-istio-token\" (UniqueName: \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-istio-token\") pod \"53147cfc-27a6-4f61-863a-732bfd063831\" (UID: \"53147cfc-27a6-4f61-863a-732bfd063831\") " Nov 01 22:47:25.819335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.819183 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-kube-api-access-8cj2g\" (UniqueName: \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-kube-api-access-8cj2g\") pod \"f1daa908-9812-4ecf-923d-160fca8742c7\" (UID: \"f1daa908-9812-4ecf-923d-160fca8742c7\") " Nov 01 22:47:25.819335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.819257 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-socket" pod "f1daa908-9812-4ecf-923d-160fca8742c7" (UID: "f1daa908-9812-4ecf-923d-160fca8742c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.819613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.819193 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-data" pod "53147cfc-27a6-4f61-863a-732bfd063831" (UID: "53147cfc-27a6-4f61-863a-732bfd063831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.819613 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.819464 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-credential-socket" pod "f1daa908-9812-4ecf-923d-160fca8742c7" (UID: "f1daa908-9812-4ecf-923d-160fca8742c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.819730 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.819675 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1daa908-9812-4ecf-923d-160fca8742c7-istiod-ca-cert" pod "f1daa908-9812-4ecf-923d-160fca8742c7" (UID: "f1daa908-9812-4ecf-923d-160fca8742c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:25.820744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.819950 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.820744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.819975 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1daa908-9812-4ecf-923d-160fca8742c7-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.820744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.819990 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.820744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.820004 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.820744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.820017 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.820744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.820032 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.820744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.820342 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-socket" pod "53147cfc-27a6-4f61-863a-732bfd063831" (UID: "53147cfc-27a6-4f61-863a-732bfd063831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.820744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.820396 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-credential-socket" pod "53147cfc-27a6-4f61-863a-732bfd063831" (UID: "53147cfc-27a6-4f61-863a-732bfd063831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.820744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.820557 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-certs" pod "f1daa908-9812-4ecf-923d-160fca8742c7" (UID: "f1daa908-9812-4ecf-923d-160fca8742c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.820744 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.820713 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53147cfc-27a6-4f61-863a-732bfd063831-istiod-ca-cert" pod "53147cfc-27a6-4f61-863a-732bfd063831" (UID: "53147cfc-27a6-4f61-863a-732bfd063831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:25.821367 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.821190 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-envoy" pod "53147cfc-27a6-4f61-863a-732bfd063831" (UID: "53147cfc-27a6-4f61-863a-732bfd063831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.821951 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.821926 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-kube-api-access-8cj2g" pod "f1daa908-9812-4ecf-923d-160fca8742c7" (UID: "f1daa908-9812-4ecf-923d-160fca8742c7"). InnerVolumeSpecName "kube-api-access-8cj2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:25.822945 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.822920 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1daa908-9812-4ecf-923d-160fca8742c7-istio-podinfo" pod "f1daa908-9812-4ecf-923d-160fca8742c7" (UID: "f1daa908-9812-4ecf-923d-160fca8742c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:25.822945 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.822927 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53147cfc-27a6-4f61-863a-732bfd063831-istio-podinfo" pod "53147cfc-27a6-4f61-863a-732bfd063831" (UID: "53147cfc-27a6-4f61-863a-732bfd063831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:25.824152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.824092 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-istio-token" pod "f1daa908-9812-4ecf-923d-160fca8742c7" (UID: "f1daa908-9812-4ecf-923d-160fca8742c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:25.824377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.824354 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-envoy" pod "f1daa908-9812-4ecf-923d-160fca8742c7" (UID: "f1daa908-9812-4ecf-923d-160fca8742c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.824482 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.824460 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-kube-api-access-hdtqt" pod "53147cfc-27a6-4f61-863a-732bfd063831" (UID: "53147cfc-27a6-4f61-863a-732bfd063831"). InnerVolumeSpecName "kube-api-access-hdtqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:25.824833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.824807 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-istio-token" pod "53147cfc-27a6-4f61-863a-732bfd063831" (UID: "53147cfc-27a6-4f61-863a-732bfd063831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:25.921620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921588 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.921620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921617 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.921620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921628 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdtqt\" (UniqueName: \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-kube-api-access-hdtqt\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.921873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921638 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.921873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921648 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53147cfc-27a6-4f61-863a-732bfd063831-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.921873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921661 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1daa908-9812-4ecf-923d-160fca8742c7-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.921873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921670 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.921873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921679 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53147cfc-27a6-4f61-863a-732bfd063831-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.921873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921687 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1daa908-9812-4ecf-923d-160fca8742c7-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.921873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921695 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53147cfc-27a6-4f61-863a-732bfd063831-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.921873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921703 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53147cfc-27a6-4f61-863a-732bfd063831-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.921873 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:25.921711 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cj2g\" (UniqueName: \"kubernetes.io/projected/f1daa908-9812-4ecf-923d-160fca8742c7-kube-api-access-8cj2g\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.068218 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.068068 6466 generic.go:356] "Generic (PLEG): container finished" podID="f1daa908-9812-4ecf-923d-160fca8742c7" containerID="151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263" exitCode=0 Nov 01 22:47:26.068218 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.068145 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" Nov 01 22:47:26.068440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.068140 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" event={"ID":"f1daa908-9812-4ecf-923d-160fca8742c7","Type":"ContainerDied","Data":"151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263"} Nov 01 22:47:26.068440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.068264 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr" event={"ID":"f1daa908-9812-4ecf-923d-160fca8742c7","Type":"ContainerDied","Data":"c0fb4863a3ee5f0090a8bb186e437d540427ae7d76150145a421f4a465a411e2"} Nov 01 22:47:26.068440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.068298 6466 scope.go:122] "RemoveContainer" containerID="151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263" Nov 01 22:47:26.070327 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.070297 6466 generic.go:356] "Generic (PLEG): container finished" podID="53147cfc-27a6-4f61-863a-732bfd063831" containerID="741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4" exitCode=0 Nov 01 22:47:26.070457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.070358 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" Nov 01 22:47:26.070457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.070383 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" event={"ID":"53147cfc-27a6-4f61-863a-732bfd063831","Type":"ContainerDied","Data":"741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4"} Nov 01 22:47:26.070457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.070430 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn" event={"ID":"53147cfc-27a6-4f61-863a-732bfd063831","Type":"ContainerDied","Data":"09d71a33dd18cd2d8d665f2e4a34496fe040287018f37bde1bd04130ae083e28"} Nov 01 22:47:26.079357 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.079333 6466 scope.go:122] "RemoveContainer" containerID="151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263" Nov 01 22:47:26.079671 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:26.079650 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263\": container with ID starting with 151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263 not found: ID does not exist" containerID="151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263" Nov 01 22:47:26.079764 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.079683 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263"} err="failed to get container status \"151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263\": rpc error: code = NotFound desc = could not find container \"151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263\": container with ID starting with 151eed74e847fc01c6b1c335958d9298fbfd3c8bb4f5cd0afd17adb368afd263 not found: ID does not exist" Nov 01 22:47:26.079764 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.079711 6466 scope.go:122] "RemoveContainer" containerID="741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4" Nov 01 22:47:26.089407 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.089386 6466 scope.go:122] "RemoveContainer" containerID="ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98" Nov 01 22:47:26.097679 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.097652 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr"] Nov 01 22:47:26.100749 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.100732 6466 scope.go:122] "RemoveContainer" containerID="741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4" Nov 01 22:47:26.101165 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:26.101096 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4\": container with ID starting with 741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4 not found: ID does not exist" containerID="741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4" Nov 01 22:47:26.101272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.101177 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4"} err="failed to get container status \"741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4\": rpc error: code = NotFound desc = could not find container \"741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4\": container with ID starting with 741892925ff49585e47031f137cd0891af478fe3e4faff984277edb7b1c843b4 not found: ID does not exist" Nov 01 22:47:26.101272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.101204 6466 scope.go:122] "RemoveContainer" containerID="ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98" Nov 01 22:47:26.101535 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:26.101508 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98\": container with ID starting with ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98 not found: ID does not exist" containerID="ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98" Nov 01 22:47:26.101644 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.101541 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98"} err="failed to get container status \"ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98\": rpc error: code = NotFound desc = could not find container \"ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98\": container with ID starting with ab1e7bc20a89be741eb019bd1fbb423ddc02a3f7427fbf1b9446c4413f265d98 not found: ID does not exist" Nov 01 22:47:26.103487 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.103469 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-9vghr"] Nov 01 22:47:26.126920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.125040 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn"] Nov 01 22:47:26.136014 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.135982 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-whtxn"] Nov 01 22:47:26.341497 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.341460 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53147cfc-27a6-4f61-863a-732bfd063831" path="/var/lib/kubelet/pods/53147cfc-27a6-4f61-863a-732bfd063831/volumes" Nov 01 22:47:26.342045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.342027 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" path="/var/lib/kubelet/pods/f1daa908-9812-4ecf-923d-160fca8742c7/volumes" Nov 01 22:47:26.531961 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.531873 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:47:26.585333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.585304 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:47:26.674131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.650649 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-kube-api-access-65bjm\" (UniqueName: \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-kube-api-access-65bjm\") pod \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " Nov 01 22:47:26.687888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.687858 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-envoy\") pod \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " Nov 01 22:47:26.688179 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.688159 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-credential-socket\") pod \"9a12f352-2870-4eb9-992f-d1d12348f299\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " Nov 01 22:47:26.688766 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.688735 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-podinfo\") pod \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " Nov 01 22:47:26.689314 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.688651 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-credential-socket" pod "9a12f352-2870-4eb9-992f-d1d12348f299" (UID: "9a12f352-2870-4eb9-992f-d1d12348f299"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.689417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.689288 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a12f352-2870-4eb9-992f-d1d12348f299-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a12f352-2870-4eb9-992f-d1d12348f299-istio-podinfo\") pod \"9a12f352-2870-4eb9-992f-d1d12348f299\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " Nov 01 22:47:26.689417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.689354 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istiod-ca-cert\") pod \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " Nov 01 22:47:26.689658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.689637 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-certs\") pod \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " Nov 01 22:47:26.689741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.689708 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a12f352-2870-4eb9-992f-d1d12348f299-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a12f352-2870-4eb9-992f-d1d12348f299-istiod-ca-cert\") pod \"9a12f352-2870-4eb9-992f-d1d12348f299\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " Nov 01 22:47:26.689807 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.689750 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-socket\") pod \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " Nov 01 22:47:26.689807 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.689787 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-credential-socket\") pod \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " Nov 01 22:47:26.689923 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.689882 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-istio-token\" (UniqueName: \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-istio-token\") pod \"9a12f352-2870-4eb9-992f-d1d12348f299\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " Nov 01 22:47:26.689984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.689921 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-data\") pod \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " Nov 01 22:47:26.689984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.689965 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-certs\") pod \"9a12f352-2870-4eb9-992f-d1d12348f299\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " Nov 01 22:47:26.690084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.690003 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-token\" (UniqueName: \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-token\") pod \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\" (UID: \"51a87e58-fa5f-4806-81cd-bed0b1e62ece\") " Nov 01 22:47:26.690084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.690077 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-envoy\") pod \"9a12f352-2870-4eb9-992f-d1d12348f299\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " Nov 01 22:47:26.690221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.690126 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-socket\") pod \"9a12f352-2870-4eb9-992f-d1d12348f299\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " Nov 01 22:47:26.690221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.690203 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-kube-api-access-fm857\" (UniqueName: \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-kube-api-access-fm857\") pod \"9a12f352-2870-4eb9-992f-d1d12348f299\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " Nov 01 22:47:26.692195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.690919 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.692195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.691204 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a12f352-2870-4eb9-992f-d1d12348f299-istiod-ca-cert" pod "9a12f352-2870-4eb9-992f-d1d12348f299" (UID: "9a12f352-2870-4eb9-992f-d1d12348f299"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:26.692195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.691606 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istiod-ca-cert" pod "51a87e58-fa5f-4806-81cd-bed0b1e62ece" (UID: "51a87e58-fa5f-4806-81cd-bed0b1e62ece"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:26.692195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.691839 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-certs" pod "51a87e58-fa5f-4806-81cd-bed0b1e62ece" (UID: "51a87e58-fa5f-4806-81cd-bed0b1e62ece"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.700521 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.700468 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-socket" pod "51a87e58-fa5f-4806-81cd-bed0b1e62ece" (UID: "51a87e58-fa5f-4806-81cd-bed0b1e62ece"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.700695 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.700617 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-kube-api-access-65bjm" pod "51a87e58-fa5f-4806-81cd-bed0b1e62ece" (UID: "51a87e58-fa5f-4806-81cd-bed0b1e62ece"). InnerVolumeSpecName "kube-api-access-65bjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:26.701472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.701281 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-envoy" pod "9a12f352-2870-4eb9-992f-d1d12348f299" (UID: "9a12f352-2870-4eb9-992f-d1d12348f299"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.701771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.701611 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-credential-socket" pod "51a87e58-fa5f-4806-81cd-bed0b1e62ece" (UID: "51a87e58-fa5f-4806-81cd-bed0b1e62ece"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.702034 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.701881 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-data" pod "51a87e58-fa5f-4806-81cd-bed0b1e62ece" (UID: "51a87e58-fa5f-4806-81cd-bed0b1e62ece"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.702858 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.702162 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-certs" pod "9a12f352-2870-4eb9-992f-d1d12348f299" (UID: "9a12f352-2870-4eb9-992f-d1d12348f299"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.704640 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.704602 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a12f352-2870-4eb9-992f-d1d12348f299-istio-podinfo" pod "9a12f352-2870-4eb9-992f-d1d12348f299" (UID: "9a12f352-2870-4eb9-992f-d1d12348f299"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:26.704975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.704935 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-istio-token" pod "9a12f352-2870-4eb9-992f-d1d12348f299" (UID: "9a12f352-2870-4eb9-992f-d1d12348f299"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:26.727984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.727206 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-socket" pod "9a12f352-2870-4eb9-992f-d1d12348f299" (UID: "9a12f352-2870-4eb9-992f-d1d12348f299"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.741948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.741905 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-envoy" pod "51a87e58-fa5f-4806-81cd-bed0b1e62ece" (UID: "51a87e58-fa5f-4806-81cd-bed0b1e62ece"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.742500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.742455 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-token" pod "51a87e58-fa5f-4806-81cd-bed0b1e62ece" (UID: "51a87e58-fa5f-4806-81cd-bed0b1e62ece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:26.742783 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.742751 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-podinfo" pod "51a87e58-fa5f-4806-81cd-bed0b1e62ece" (UID: "51a87e58-fa5f-4806-81cd-bed0b1e62ece"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:26.744898 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.744863 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-kube-api-access-fm857" pod "9a12f352-2870-4eb9-992f-d1d12348f299" (UID: "9a12f352-2870-4eb9-992f-d1d12348f299"). InnerVolumeSpecName "kube-api-access-fm857". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:26.791915 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.791833 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-data\") pod \"9a12f352-2870-4eb9-992f-d1d12348f299\" (UID: \"9a12f352-2870-4eb9-992f-d1d12348f299\") " Nov 01 22:47:26.792219 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792184 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-data" pod "9a12f352-2870-4eb9-992f-d1d12348f299" (UID: "9a12f352-2870-4eb9-992f-d1d12348f299"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.792612 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792587 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792619 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a12f352-2870-4eb9-992f-d1d12348f299-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792635 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792649 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792669 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792683 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792697 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792709 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792721 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792735 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792748 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm857\" (UniqueName: \"kubernetes.io/projected/9a12f352-2870-4eb9-992f-d1d12348f299-kube-api-access-fm857\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792764 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65bjm\" (UniqueName: \"kubernetes.io/projected/51a87e58-fa5f-4806-81cd-bed0b1e62ece-kube-api-access-65bjm\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792779 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a12f352-2870-4eb9-992f-d1d12348f299-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792792 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792807 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792820 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a12f352-2870-4eb9-992f-d1d12348f299-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.792984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:26.792833 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a87e58-fa5f-4806-81cd-bed0b1e62ece-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:27.100084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.099998 6466 generic.go:356] "Generic (PLEG): container finished" podID="9a12f352-2870-4eb9-992f-d1d12348f299" containerID="70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84" exitCode=0 Nov 01 22:47:27.100084 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.100046 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" Nov 01 22:47:27.100359 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.100087 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" event={"ID":"9a12f352-2870-4eb9-992f-d1d12348f299","Type":"ContainerDied","Data":"70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84"} Nov 01 22:47:27.100359 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.100156 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx" event={"ID":"9a12f352-2870-4eb9-992f-d1d12348f299","Type":"ContainerDied","Data":"ec2f18fea57c40a60e0257bc820f6e2d734dc18511fea4a416450c832d5f6f01"} Nov 01 22:47:27.100359 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.100190 6466 scope.go:122] "RemoveContainer" containerID="70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84" Nov 01 22:47:27.106955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.106931 6466 generic.go:356] "Generic (PLEG): container finished" podID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerID="281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b" exitCode=0 Nov 01 22:47:27.107065 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.107051 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" Nov 01 22:47:27.108801 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.108563 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" event={"ID":"51a87e58-fa5f-4806-81cd-bed0b1e62ece","Type":"ContainerDied","Data":"281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b"} Nov 01 22:47:27.108801 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.108616 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm" event={"ID":"51a87e58-fa5f-4806-81cd-bed0b1e62ece","Type":"ContainerDied","Data":"26465d2338bc2879481c0dcbfca11b060645b14e855279b1399f2089f0b54eee"} Nov 01 22:47:27.118493 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.118343 6466 scope.go:122] "RemoveContainer" containerID="bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8" Nov 01 22:47:27.128396 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.128371 6466 scope.go:122] "RemoveContainer" containerID="70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84" Nov 01 22:47:27.128746 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:27.128694 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84\": container with ID starting with 70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84 not found: ID does not exist" containerID="70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84" Nov 01 22:47:27.128746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.128726 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84"} err="failed to get container status \"70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84\": rpc error: code = NotFound desc = could not find container \"70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84\": container with ID starting with 70091c531d6f10827d3b8b68f6a8580c6942cfc9eb9efe9594fd102f34334b84 not found: ID does not exist" Nov 01 22:47:27.128889 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.128748 6466 scope.go:122] "RemoveContainer" containerID="bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8" Nov 01 22:47:27.129079 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:27.129044 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8\": container with ID starting with bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8 not found: ID does not exist" containerID="bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8" Nov 01 22:47:27.129702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.129079 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8"} err="failed to get container status \"bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8\": rpc error: code = NotFound desc = could not find container \"bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8\": container with ID starting with bf6bbdafa70b2d733d3a642891919446db7d252f1bfe5424f9c1957ba10cd9f8 not found: ID does not exist" Nov 01 22:47:27.129702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.129101 6466 scope.go:122] "RemoveContainer" containerID="281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b" Nov 01 22:47:27.133490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.133469 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx"] Nov 01 22:47:27.140513 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.140494 6466 scope.go:122] "RemoveContainer" containerID="281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b" Nov 01 22:47:27.140807 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:27.140784 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b\": container with ID starting with 281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b not found: ID does not exist" containerID="281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b" Nov 01 22:47:27.140880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.140818 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b"} err="failed to get container status \"281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b\": rpc error: code = NotFound desc = could not find container \"281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b\": container with ID starting with 281fd6082918e46b3dfd7b325821db5dc9ae3d41be086506c457876b8e93f28b not found: ID does not exist" Nov 01 22:47:27.141668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.141650 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-gwqhx"] Nov 01 22:47:27.162382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.162353 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm"] Nov 01 22:47:27.165987 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:27.165964 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-b6dxm"] Nov 01 22:47:28.343121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:28.343068 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" path="/var/lib/kubelet/pods/51a87e58-fa5f-4806-81cd-bed0b1e62ece/volumes" Nov 01 22:47:28.343668 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:28.343651 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" path="/var/lib/kubelet/pods/9a12f352-2870-4eb9-992f-d1d12348f299/volumes" Nov 01 22:47:30.096443 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:30.096362 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" podUID="67394475-a855-440e-b6bd-8efb1ce7a539" containerName="payload-processing" containerID="cri-o://e4bf276dd5d6b0ab312ecbd8178fa9235b8baa96bb27e1527bc1dd198a2deaec" gracePeriod=25 Nov 01 22:47:30.136736 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:30.136697 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" podUID="6c8a6a1e-70e8-4c0a-9a72-ca58a711841d" containerName="payload-pre-processing" containerID="cri-o://ca097437a44483cb2e20571615b0d3112d8288d3c4b07a6de1da5146c80b1287" gracePeriod=25 Nov 01 22:47:33.640720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:33.640683 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" podUID="6c8a6a1e-70e8-4c0a-9a72-ca58a711841d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:47:33.641099 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:33.641074 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" podUID="67394475-a855-440e-b6bd-8efb1ce7a539" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:47:34.961366 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:34.961326 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" containerID="cri-o://a0a0d219e3186e6aec22586d96212ab6d8d7f602330b3dbfdda1ef104f57f401" gracePeriod=45 Nov 01 22:47:35.180306 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.180278 6466 generic.go:356] "Generic (PLEG): container finished" podID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerID="a0a0d219e3186e6aec22586d96212ab6d8d7f602330b3dbfdda1ef104f57f401" exitCode=0 Nov 01 22:47:35.180573 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.180360 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" event={"ID":"024bf9b0-d2dd-4499-a45b-043af8b73da2","Type":"ContainerDied","Data":"a0a0d219e3186e6aec22586d96212ab6d8d7f602330b3dbfdda1ef104f57f401"} Nov 01 22:47:35.248876 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.248834 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:47:35.324044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324017 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-home\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-home\") pod \"024bf9b0-d2dd-4499-a45b-043af8b73da2\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " Nov 01 22:47:35.324221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324089 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024bf9b0-d2dd-4499-a45b-043af8b73da2-tls-certs\" (UniqueName: \"kubernetes.io/secret/024bf9b0-d2dd-4499-a45b-043af8b73da2-tls-certs\") pod \"024bf9b0-d2dd-4499-a45b-043af8b73da2\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " Nov 01 22:47:35.324221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324168 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-tmp-dir\") pod \"024bf9b0-d2dd-4499-a45b-043af8b73da2\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " Nov 01 22:47:35.324221 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324205 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-dshm\") pod \"024bf9b0-d2dd-4499-a45b-043af8b73da2\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " Nov 01 22:47:35.324362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324224 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-model-cache\") pod \"024bf9b0-d2dd-4499-a45b-043af8b73da2\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " Nov 01 22:47:35.324362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324260 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024bf9b0-d2dd-4499-a45b-043af8b73da2-kube-api-access-8txbl\" (UniqueName: \"kubernetes.io/projected/024bf9b0-d2dd-4499-a45b-043af8b73da2-kube-api-access-8txbl\") pod \"024bf9b0-d2dd-4499-a45b-043af8b73da2\" (UID: \"024bf9b0-d2dd-4499-a45b-043af8b73da2\") " Nov 01 22:47:35.324362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324325 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-home" pod "024bf9b0-d2dd-4499-a45b-043af8b73da2" (UID: "024bf9b0-d2dd-4499-a45b-043af8b73da2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:35.324476 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324406 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-tmp-dir" pod "024bf9b0-d2dd-4499-a45b-043af8b73da2" (UID: "024bf9b0-d2dd-4499-a45b-043af8b73da2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:35.324581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324556 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-model-cache" pod "024bf9b0-d2dd-4499-a45b-043af8b73da2" (UID: "024bf9b0-d2dd-4499-a45b-043af8b73da2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:35.324980 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324961 6466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-tmp-dir\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:35.325059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324984 6466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-model-cache\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:35.325059 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.324999 6466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-home\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:35.326410 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.326383 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-dshm" pod "024bf9b0-d2dd-4499-a45b-043af8b73da2" (UID: "024bf9b0-d2dd-4499-a45b-043af8b73da2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:35.326492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.326432 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024bf9b0-d2dd-4499-a45b-043af8b73da2-tls-certs" pod "024bf9b0-d2dd-4499-a45b-043af8b73da2" (UID: "024bf9b0-d2dd-4499-a45b-043af8b73da2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:35.326492 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.326437 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024bf9b0-d2dd-4499-a45b-043af8b73da2-kube-api-access-8txbl" pod "024bf9b0-d2dd-4499-a45b-043af8b73da2" (UID: "024bf9b0-d2dd-4499-a45b-043af8b73da2"). InnerVolumeSpecName "kube-api-access-8txbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:35.434522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.434439 6466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024bf9b0-d2dd-4499-a45b-043af8b73da2-dshm\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:35.434522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.434479 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8txbl\" (UniqueName: \"kubernetes.io/projected/024bf9b0-d2dd-4499-a45b-043af8b73da2-kube-api-access-8txbl\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:35.434522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:35.434494 6466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024bf9b0-d2dd-4499-a45b-043af8b73da2-tls-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:36.189479 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:36.189437 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" event={"ID":"024bf9b0-d2dd-4499-a45b-043af8b73da2","Type":"ContainerDied","Data":"80f4ab4a4bd6e3124e249e245514067de50c6c03205a681f1fb473bd4a9c7265"} Nov 01 22:47:36.189479 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:36.189475 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq" Nov 01 22:47:36.190050 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:36.189503 6466 scope.go:122] "RemoveContainer" containerID="a0a0d219e3186e6aec22586d96212ab6d8d7f602330b3dbfdda1ef104f57f401" Nov 01 22:47:36.224468 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:36.224434 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq"] Nov 01 22:47:36.233342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:36.233176 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-2-w6-b28a40-kserve-8666c65fc9-bpwgq"] Nov 01 22:47:36.342915 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:36.342865 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" path="/var/lib/kubelet/pods/024bf9b0-d2dd-4499-a45b-043af8b73da2/volumes" Nov 01 22:47:43.640319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:43.640284 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" podUID="67394475-a855-440e-b6bd-8efb1ce7a539" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:47:43.640807 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:43.640484 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" podUID="6c8a6a1e-70e8-4c0a-9a72-ca58a711841d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:47:48.551813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:48.551773 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6"] Nov 01 22:47:48.552316 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:48.552031 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" podUID="ff764f93-2273-4694-bb85-72533fc7161c" containerName="maas-api" containerID="cri-o://f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d" gracePeriod=30 Nov 01 22:47:48.951998 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:48.951695 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:47:49.116621 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.104986 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-metrics-certs\") pod \"ff764f93-2273-4694-bb85-72533fc7161c\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " Nov 01 22:47:49.116621 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.105161 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff764f93-2273-4694-bb85-72533fc7161c-kube-api-access-9qjk8\" (UniqueName: \"kubernetes.io/projected/ff764f93-2273-4694-bb85-72533fc7161c-kube-api-access-9qjk8\") pod \"ff764f93-2273-4694-bb85-72533fc7161c\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " Nov 01 22:47:49.116621 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.105315 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-maas-api-tls\") pod \"ff764f93-2273-4694-bb85-72533fc7161c\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " Nov 01 22:47:49.116621 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.105400 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff764f93-2273-4694-bb85-72533fc7161c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff764f93-2273-4694-bb85-72533fc7161c-openshift-service-ca\") pod \"ff764f93-2273-4694-bb85-72533fc7161c\" (UID: \"ff764f93-2273-4694-bb85-72533fc7161c\") " Nov 01 22:47:49.126702 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.126639 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff764f93-2273-4694-bb85-72533fc7161c-openshift-service-ca" pod "ff764f93-2273-4694-bb85-72533fc7161c" (UID: "ff764f93-2273-4694-bb85-72533fc7161c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:49.127909 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.127863 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-metrics-certs" pod "ff764f93-2273-4694-bb85-72533fc7161c" (UID: "ff764f93-2273-4694-bb85-72533fc7161c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:49.129039 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.128993 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-maas-api-tls" pod "ff764f93-2273-4694-bb85-72533fc7161c" (UID: "ff764f93-2273-4694-bb85-72533fc7161c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:49.129568 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.129541 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff764f93-2273-4694-bb85-72533fc7161c-kube-api-access-9qjk8" pod "ff764f93-2273-4694-bb85-72533fc7161c" (UID: "ff764f93-2273-4694-bb85-72533fc7161c"). InnerVolumeSpecName "kube-api-access-9qjk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:49.206925 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.206898 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff764f93-2273-4694-bb85-72533fc7161c-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:49.206925 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.206923 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:49.207141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.206934 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qjk8\" (UniqueName: \"kubernetes.io/projected/ff764f93-2273-4694-bb85-72533fc7161c-kube-api-access-9qjk8\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:49.207141 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.206946 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff764f93-2273-4694-bb85-72533fc7161c-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:49.282001 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:49.281962 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:47:49.392255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.391200 6466 generic.go:356] "Generic (PLEG): container finished" podID="ff764f93-2273-4694-bb85-72533fc7161c" containerID="f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d" exitCode=0 Nov 01 22:47:49.392255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.391389 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" event={"ID":"ff764f93-2273-4694-bb85-72533fc7161c","Type":"ContainerDied","Data":"f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d"} Nov 01 22:47:49.392255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.391446 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" event={"ID":"ff764f93-2273-4694-bb85-72533fc7161c","Type":"ContainerDied","Data":"9bb6878005ea36a56bf4e679ab2af7c64e60979aac7e350a4a1a70d76f209d1e"} Nov 01 22:47:49.392255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.391475 6466 scope.go:122] "RemoveContainer" containerID="f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d" Nov 01 22:47:49.392255 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.391632 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6" Nov 01 22:47:49.439542 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.439507 6466 scope.go:122] "RemoveContainer" containerID="f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d" Nov 01 22:47:49.440154 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:47:49.440065 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d\": container with ID starting with f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d not found: ID does not exist" containerID="f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d" Nov 01 22:47:49.440377 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.440343 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d"} err="failed to get container status \"f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d\": rpc error: code = NotFound desc = could not find container \"f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d\": container with ID starting with f3153635d9766a1658a661b7b834e9fde734b0d6d3310a83a1288edcd84cc04d not found: ID does not exist" Nov 01 22:47:49.460296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.457310 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6"] Nov 01 22:47:49.460296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:49.459922 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b28a40-7ff5658dd4-xthn6"] Nov 01 22:47:50.346522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.346488 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff764f93-2273-4694-bb85-72533fc7161c" path="/var/lib/kubelet/pods/ff764f93-2273-4694-bb85-72533fc7161c/volumes" Nov 01 22:47:50.416103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.416070 6466 generic.go:356] "Generic (PLEG): container finished" podID="6c8a6a1e-70e8-4c0a-9a72-ca58a711841d" containerID="ca097437a44483cb2e20571615b0d3112d8288d3c4b07a6de1da5146c80b1287" exitCode=0 Nov 01 22:47:50.416282 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.416171 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" event={"ID":"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d","Type":"ContainerDied","Data":"ca097437a44483cb2e20571615b0d3112d8288d3c4b07a6de1da5146c80b1287"} Nov 01 22:47:50.419206 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.419174 6466 generic.go:356] "Generic (PLEG): container finished" podID="67394475-a855-440e-b6bd-8efb1ce7a539" containerID="e4bf276dd5d6b0ab312ecbd8178fa9235b8baa96bb27e1527bc1dd198a2deaec" exitCode=0 Nov 01 22:47:50.419402 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.419380 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" event={"ID":"67394475-a855-440e-b6bd-8efb1ce7a539","Type":"ContainerDied","Data":"e4bf276dd5d6b0ab312ecbd8178fa9235b8baa96bb27e1527bc1dd198a2deaec"} Nov 01 22:47:50.419483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.419433 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" event={"ID":"67394475-a855-440e-b6bd-8efb1ce7a539","Type":"ContainerDied","Data":"20a83b8c3b0e3c957d627a12a5c57cf74f60897eed847c1741bc5f783652e25b"} Nov 01 22:47:50.419483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.419459 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20a83b8c3b0e3c957d627a12a5c57cf74f60897eed847c1741bc5f783652e25b" Nov 01 22:47:50.423804 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.423782 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:47:50.466897 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.466871 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:47:50.539735 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.539704 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-plugins-config-volume\") pod \"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d\" (UID: \"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d\") " Nov 01 22:47:50.539936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.539798 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67394475-a855-440e-b6bd-8efb1ce7a539-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67394475-a855-440e-b6bd-8efb1ce7a539-plugins-config-volume\") pod \"67394475-a855-440e-b6bd-8efb1ce7a539\" (UID: \"67394475-a855-440e-b6bd-8efb1ce7a539\") " Nov 01 22:47:50.539936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.539890 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67394475-a855-440e-b6bd-8efb1ce7a539-kube-api-access-k9jsf\" (UniqueName: \"kubernetes.io/projected/67394475-a855-440e-b6bd-8efb1ce7a539-kube-api-access-k9jsf\") pod \"67394475-a855-440e-b6bd-8efb1ce7a539\" (UID: \"67394475-a855-440e-b6bd-8efb1ce7a539\") " Nov 01 22:47:50.540101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.540067 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-kube-api-access-8p2wx\" (UniqueName: \"kubernetes.io/projected/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-kube-api-access-8p2wx\") pod \"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d\" (UID: \"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d\") " Nov 01 22:47:50.540211 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.540185 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-plugins-config-volume" pod "6c8a6a1e-70e8-4c0a-9a72-ca58a711841d" (UID: "6c8a6a1e-70e8-4c0a-9a72-ca58a711841d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:50.540339 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.540312 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67394475-a855-440e-b6bd-8efb1ce7a539-plugins-config-volume" pod "67394475-a855-440e-b6bd-8efb1ce7a539" (UID: "67394475-a855-440e-b6bd-8efb1ce7a539"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:50.540790 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.540761 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67394475-a855-440e-b6bd-8efb1ce7a539-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:50.540909 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.540794 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:50.542317 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.542295 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67394475-a855-440e-b6bd-8efb1ce7a539-kube-api-access-k9jsf" pod "67394475-a855-440e-b6bd-8efb1ce7a539" (UID: "67394475-a855-440e-b6bd-8efb1ce7a539"). InnerVolumeSpecName "kube-api-access-k9jsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:50.542798 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.542774 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-kube-api-access-8p2wx" pod "6c8a6a1e-70e8-4c0a-9a72-ca58a711841d" (UID: "6c8a6a1e-70e8-4c0a-9a72-ca58a711841d"). InnerVolumeSpecName "kube-api-access-8p2wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:50.642242 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.642207 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9jsf\" (UniqueName: \"kubernetes.io/projected/67394475-a855-440e-b6bd-8efb1ce7a539-kube-api-access-k9jsf\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:50.642242 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:50.642237 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p2wx\" (UniqueName: \"kubernetes.io/projected/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d-kube-api-access-8p2wx\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:51.428554 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:51.428521 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" Nov 01 22:47:51.428986 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:51.428570 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl" event={"ID":"6c8a6a1e-70e8-4c0a-9a72-ca58a711841d","Type":"ContainerDied","Data":"0195f83f74c3fbd555b202daf5cdc5973170412445bfe5c413e09c123ac7fae7"} Nov 01 22:47:51.428986 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:51.428627 6466 scope.go:122] "RemoveContainer" containerID="ca097437a44483cb2e20571615b0d3112d8288d3c4b07a6de1da5146c80b1287" Nov 01 22:47:51.428986 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:51.428705 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4" Nov 01 22:47:51.456312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:51.456288 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4"] Nov 01 22:47:51.458697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:51.458677 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b28a40-86f7d87c8f-ncvq4"] Nov 01 22:47:51.468441 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:51.468420 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl"] Nov 01 22:47:51.473657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:51.473633 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b28a40-5f89f5897f-dlnrl"] Nov 01 22:47:52.342184 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:52.342142 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67394475-a855-440e-b6bd-8efb1ce7a539" path="/var/lib/kubelet/pods/67394475-a855-440e-b6bd-8efb1ce7a539/volumes" Nov 01 22:47:52.342683 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:52.342665 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8a6a1e-70e8-4c0a-9a72-ca58a711841d" path="/var/lib/kubelet/pods/6c8a6a1e-70e8-4c0a-9a72-ca58a711841d/volumes" Nov 01 22:47:57.133341 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:57.133307 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt"] Nov 01 22:47:57.134088 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:57.134053 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" containerID="cri-o://63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee" gracePeriod=30 Nov 01 22:47:57.137592 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:57.137562 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg"] Nov 01 22:47:57.137979 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:57.137929 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" containerID="cri-o://309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3" gracePeriod=30 Nov 01 22:47:57.149684 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:57.149656 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt"] Nov 01 22:47:57.149932 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:47:57.149906 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" containerID="cri-o://cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679" gracePeriod=30 Nov 01 22:48:02.443800 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.443764 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:48:02.459261 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.459226 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:48:02.529070 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.529040 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:48:02.537163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537135 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-certs\") pod \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " Nov 01 22:48:02.537293 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537245 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-podinfo\") pod \"8831591d-4318-4b8d-b6bc-9790a73d8c67\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " Nov 01 22:48:02.537293 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537278 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-token\" (UniqueName: \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-token\") pod \"8831591d-4318-4b8d-b6bc-9790a73d8c67\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " Nov 01 22:48:02.537459 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537307 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8dc27f4-6212-4492-a1d5-dcced032e3af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8dc27f4-6212-4492-a1d5-dcced032e3af-istiod-ca-cert\") pod \"b8dc27f4-6212-4492-a1d5-dcced032e3af\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " Nov 01 22:48:02.537459 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537335 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-certs" pod "fdbec9fb-83d1-405b-bbcd-bc0f30257644" (UID: "fdbec9fb-83d1-405b-bbcd-bc0f30257644"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.537459 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537341 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8831591d-4318-4b8d-b6bc-9790a73d8c67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8831591d-4318-4b8d-b6bc-9790a73d8c67-istiod-ca-cert\") pod \"8831591d-4318-4b8d-b6bc-9790a73d8c67\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " Nov 01 22:48:02.537459 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537378 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-data\") pod \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " Nov 01 22:48:02.537459 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537406 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-kube-api-access-dx7cf\" (UniqueName: \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-kube-api-access-dx7cf\") pod \"8831591d-4318-4b8d-b6bc-9790a73d8c67\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " Nov 01 22:48:02.537459 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537439 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-credential-socket\") pod \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " Nov 01 22:48:02.537794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537470 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-certs\") pod \"8831591d-4318-4b8d-b6bc-9790a73d8c67\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " Nov 01 22:48:02.537794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537520 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-socket\") pod \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " Nov 01 22:48:02.537794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537567 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-token\" (UniqueName: \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-token\") pod \"b8dc27f4-6212-4492-a1d5-dcced032e3af\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " Nov 01 22:48:02.537794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537620 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-podinfo\") pod \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " Nov 01 22:48:02.537794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537652 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-data\") pod \"b8dc27f4-6212-4492-a1d5-dcced032e3af\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " Nov 01 22:48:02.537794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537680 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-kube-api-access-6xtjl\" (UniqueName: \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-kube-api-access-6xtjl\") pod \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " Nov 01 22:48:02.537794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537709 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-podinfo\") pod \"b8dc27f4-6212-4492-a1d5-dcced032e3af\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " Nov 01 22:48:02.537794 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537747 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-envoy\") pod \"b8dc27f4-6212-4492-a1d5-dcced032e3af\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " Nov 01 22:48:02.538263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537812 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-credential-socket\") pod \"8831591d-4318-4b8d-b6bc-9790a73d8c67\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " Nov 01 22:48:02.538263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537819 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-data" pod "fdbec9fb-83d1-405b-bbcd-bc0f30257644" (UID: "fdbec9fb-83d1-405b-bbcd-bc0f30257644"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.538263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537848 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-data\") pod \"8831591d-4318-4b8d-b6bc-9790a73d8c67\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " Nov 01 22:48:02.538263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537877 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-socket\") pod \"b8dc27f4-6212-4492-a1d5-dcced032e3af\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " Nov 01 22:48:02.538263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537915 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-envoy\") pod \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " Nov 01 22:48:02.538263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537953 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istiod-ca-cert\") pod \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " Nov 01 22:48:02.538263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.537991 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-credential-socket\") pod \"b8dc27f4-6212-4492-a1d5-dcced032e3af\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " Nov 01 22:48:02.538263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538048 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-credential-socket" pod "fdbec9fb-83d1-405b-bbcd-bc0f30257644" (UID: "fdbec9fb-83d1-405b-bbcd-bc0f30257644"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.538263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538057 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-socket\") pod \"8831591d-4318-4b8d-b6bc-9790a73d8c67\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " Nov 01 22:48:02.538263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538134 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-token\" (UniqueName: \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-token\") pod \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\" (UID: \"fdbec9fb-83d1-405b-bbcd-bc0f30257644\") " Nov 01 22:48:02.538755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538278 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-socket" pod "8831591d-4318-4b8d-b6bc-9790a73d8c67" (UID: "8831591d-4318-4b8d-b6bc-9790a73d8c67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.538755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538316 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-certs\") pod \"b8dc27f4-6212-4492-a1d5-dcced032e3af\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " Nov 01 22:48:02.538755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538323 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8831591d-4318-4b8d-b6bc-9790a73d8c67-istiod-ca-cert" pod "8831591d-4318-4b8d-b6bc-9790a73d8c67" (UID: "8831591d-4318-4b8d-b6bc-9790a73d8c67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:02.538755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538341 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-socket" pod "fdbec9fb-83d1-405b-bbcd-bc0f30257644" (UID: "fdbec9fb-83d1-405b-bbcd-bc0f30257644"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.538755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538354 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-envoy\") pod \"8831591d-4318-4b8d-b6bc-9790a73d8c67\" (UID: \"8831591d-4318-4b8d-b6bc-9790a73d8c67\") " Nov 01 22:48:02.538755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538402 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-kube-api-access-29n8t\" (UniqueName: \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-kube-api-access-29n8t\") pod \"b8dc27f4-6212-4492-a1d5-dcced032e3af\" (UID: \"b8dc27f4-6212-4492-a1d5-dcced032e3af\") " Nov 01 22:48:02.538755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538470 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8dc27f4-6212-4492-a1d5-dcced032e3af-istiod-ca-cert" pod "b8dc27f4-6212-4492-a1d5-dcced032e3af" (UID: "b8dc27f4-6212-4492-a1d5-dcced032e3af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:02.538755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538484 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-data" pod "b8dc27f4-6212-4492-a1d5-dcced032e3af" (UID: "b8dc27f4-6212-4492-a1d5-dcced032e3af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.538755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538598 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-certs" pod "8831591d-4318-4b8d-b6bc-9790a73d8c67" (UID: "8831591d-4318-4b8d-b6bc-9790a73d8c67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.538755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538739 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-certs" pod "b8dc27f4-6212-4492-a1d5-dcced032e3af" (UID: "b8dc27f4-6212-4492-a1d5-dcced032e3af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.539295 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.538835 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-data" pod "8831591d-4318-4b8d-b6bc-9790a73d8c67" (UID: "8831591d-4318-4b8d-b6bc-9790a73d8c67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.539295 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.539237 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.539295 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.539259 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.539295 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.539274 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8dc27f4-6212-4492-a1d5-dcced032e3af-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.539295 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.539290 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8831591d-4318-4b8d-b6bc-9790a73d8c67-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.539457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.539303 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.539457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.539318 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.539457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.539332 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.539457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.539346 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.539457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.539360 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.539457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.539373 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.539457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.539387 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.540529 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.540303 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-credential-socket" pod "8831591d-4318-4b8d-b6bc-9790a73d8c67" (UID: "8831591d-4318-4b8d-b6bc-9790a73d8c67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.540529 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.540352 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-socket" pod "b8dc27f4-6212-4492-a1d5-dcced032e3af" (UID: "b8dc27f4-6212-4492-a1d5-dcced032e3af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.540813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.540773 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-credential-socket" pod "b8dc27f4-6212-4492-a1d5-dcced032e3af" (UID: "b8dc27f4-6212-4492-a1d5-dcced032e3af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.540927 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.540904 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istiod-ca-cert" pod "fdbec9fb-83d1-405b-bbcd-bc0f30257644" (UID: "fdbec9fb-83d1-405b-bbcd-bc0f30257644"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:02.541637 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.541602 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-token" pod "fdbec9fb-83d1-405b-bbcd-bc0f30257644" (UID: "fdbec9fb-83d1-405b-bbcd-bc0f30257644"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:02.541844 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.541811 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-podinfo" pod "b8dc27f4-6212-4492-a1d5-dcced032e3af" (UID: "b8dc27f4-6212-4492-a1d5-dcced032e3af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:48:02.542268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.542234 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-kube-api-access-6xtjl" pod "fdbec9fb-83d1-405b-bbcd-bc0f30257644" (UID: "fdbec9fb-83d1-405b-bbcd-bc0f30257644"). InnerVolumeSpecName "kube-api-access-6xtjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:02.542672 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.542650 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-envoy" pod "b8dc27f4-6212-4492-a1d5-dcced032e3af" (UID: "b8dc27f4-6212-4492-a1d5-dcced032e3af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.542845 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.542828 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-token" pod "b8dc27f4-6212-4492-a1d5-dcced032e3af" (UID: "b8dc27f4-6212-4492-a1d5-dcced032e3af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:02.542921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.542845 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-envoy" pod "8831591d-4318-4b8d-b6bc-9790a73d8c67" (UID: "8831591d-4318-4b8d-b6bc-9790a73d8c67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.543454 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.543428 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-kube-api-access-dx7cf" pod "8831591d-4318-4b8d-b6bc-9790a73d8c67" (UID: "8831591d-4318-4b8d-b6bc-9790a73d8c67"). InnerVolumeSpecName "kube-api-access-dx7cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:02.543956 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.543922 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-token" pod "8831591d-4318-4b8d-b6bc-9790a73d8c67" (UID: "8831591d-4318-4b8d-b6bc-9790a73d8c67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:02.544158 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.544044 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-kube-api-access-29n8t" pod "b8dc27f4-6212-4492-a1d5-dcced032e3af" (UID: "b8dc27f4-6212-4492-a1d5-dcced032e3af"). InnerVolumeSpecName "kube-api-access-29n8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:02.544290 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.544262 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-envoy" pod "fdbec9fb-83d1-405b-bbcd-bc0f30257644" (UID: "fdbec9fb-83d1-405b-bbcd-bc0f30257644"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.544973 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.544949 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-podinfo" pod "fdbec9fb-83d1-405b-bbcd-bc0f30257644" (UID: "fdbec9fb-83d1-405b-bbcd-bc0f30257644"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:48:02.545163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.545145 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-podinfo" pod "8831591d-4318-4b8d-b6bc-9790a73d8c67" (UID: "8831591d-4318-4b8d-b6bc-9790a73d8c67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:48:02.576771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.576735 6466 generic.go:356] "Generic (PLEG): container finished" podID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerID="cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679" exitCode=0 Nov 01 22:48:02.576966 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.576925 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" event={"ID":"fdbec9fb-83d1-405b-bbcd-bc0f30257644","Type":"ContainerDied","Data":"cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679"} Nov 01 22:48:02.577034 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.576983 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" event={"ID":"fdbec9fb-83d1-405b-bbcd-bc0f30257644","Type":"ContainerDied","Data":"19078a17652599509c5b937e7e1a504deb708fb5a4fe4db98ede9dd04b4f1a72"} Nov 01 22:48:02.577034 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.577014 6466 scope.go:122] "RemoveContainer" containerID="cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679" Nov 01 22:48:02.578166 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.577216 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt" Nov 01 22:48:02.580030 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.580001 6466 generic.go:356] "Generic (PLEG): container finished" podID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerID="63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee" exitCode=0 Nov 01 22:48:02.580153 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.580049 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" Nov 01 22:48:02.580334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.580064 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" event={"ID":"b8dc27f4-6212-4492-a1d5-dcced032e3af","Type":"ContainerDied","Data":"63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee"} Nov 01 22:48:02.580409 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.580393 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt" event={"ID":"b8dc27f4-6212-4492-a1d5-dcced032e3af","Type":"ContainerDied","Data":"2d7862b6d60f377b0378a08ed3815472caa988c6ee4c38b58c02ed094fc57115"} Nov 01 22:48:02.585172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.585145 6466 generic.go:356] "Generic (PLEG): container finished" podID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerID="309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3" exitCode=0 Nov 01 22:48:02.585306 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.585282 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" event={"ID":"8831591d-4318-4b8d-b6bc-9790a73d8c67","Type":"ContainerDied","Data":"309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3"} Nov 01 22:48:02.585443 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.585347 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" event={"ID":"8831591d-4318-4b8d-b6bc-9790a73d8c67","Type":"ContainerDied","Data":"3cb6181342526f275e73e4c6d76010330d9d6cc719cc8fff787b32d0ca74f4f8"} Nov 01 22:48:02.585443 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.585433 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg" Nov 01 22:48:02.589946 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.589927 6466 scope.go:122] "RemoveContainer" containerID="cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679" Nov 01 22:48:02.590426 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:48:02.590397 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679\": container with ID starting with cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679 not found: ID does not exist" containerID="cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679" Nov 01 22:48:02.590535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.590437 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679"} err="failed to get container status \"cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679\": rpc error: code = NotFound desc = could not find container \"cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679\": container with ID starting with cbf3e1e5c916a880531336580bf49773c00054553b25e42278057cd0f3e08679 not found: ID does not exist" Nov 01 22:48:02.590535 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.590463 6466 scope.go:122] "RemoveContainer" containerID="63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee" Nov 01 22:48:02.604865 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.604848 6466 scope.go:122] "RemoveContainer" containerID="63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee" Nov 01 22:48:02.605234 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:48:02.605217 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee\": container with ID starting with 63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee not found: ID does not exist" containerID="63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee" Nov 01 22:48:02.605295 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.605242 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee"} err="failed to get container status \"63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee\": rpc error: code = NotFound desc = could not find container \"63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee\": container with ID starting with 63dc75abc79f838c02388a60d2f4a3c9c11a5db976f509b7c4a8f631a43e5cee not found: ID does not exist" Nov 01 22:48:02.605295 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.605263 6466 scope.go:122] "RemoveContainer" containerID="309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3" Nov 01 22:48:02.618580 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.618543 6466 scope.go:122] "RemoveContainer" containerID="3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593" Nov 01 22:48:02.640887 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.640857 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.640995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.640942 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29n8t\" (UniqueName: \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-kube-api-access-29n8t\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.640995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.640961 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.640995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.640976 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.640995 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.640989 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx7cf\" (UniqueName: \"kubernetes.io/projected/8831591d-4318-4b8d-b6bc-9790a73d8c67-kube-api-access-dx7cf\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641002 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641016 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641030 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xtjl\" (UniqueName: \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-kube-api-access-6xtjl\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641043 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641056 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641069 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8831591d-4318-4b8d-b6bc-9790a73d8c67-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641083 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641096 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641164 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641154 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641168 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8dc27f4-6212-4492-a1d5-dcced032e3af-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641181 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbec9fb-83d1-405b-bbcd-bc0f30257644-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.641509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641308 6466 scope.go:122] "RemoveContainer" containerID="309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3" Nov 01 22:48:02.641509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641468 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg"] Nov 01 22:48:02.641712 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:48:02.641694 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3\": container with ID starting with 309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3 not found: ID does not exist" containerID="309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3" Nov 01 22:48:02.641749 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641719 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3"} err="failed to get container status \"309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3\": rpc error: code = NotFound desc = could not find container \"309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3\": container with ID starting with 309784b4d9a52e02d0115db5a37aa914922d1328d6226090edf50e00b9edafb3 not found: ID does not exist" Nov 01 22:48:02.641749 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.641736 6466 scope.go:122] "RemoveContainer" containerID="3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593" Nov 01 22:48:02.642060 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:48:02.642035 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593\": container with ID starting with 3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593 not found: ID does not exist" containerID="3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593" Nov 01 22:48:02.642165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.642071 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593"} err="failed to get container status \"3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593\": rpc error: code = NotFound desc = could not find container \"3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593\": container with ID starting with 3c3908b8e01b224f3c432f09f457d9c35efbae951ab76c7a722c0c6745bb1593 not found: ID does not exist" Nov 01 22:48:02.655461 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.655422 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-n5gvg"] Nov 01 22:48:02.670723 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.670642 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt"] Nov 01 22:48:02.676301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.676275 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-v44rt"] Nov 01 22:48:02.690118 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.690083 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt"] Nov 01 22:48:02.702916 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:02.702887 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-z4xdt"] Nov 01 22:48:04.341637 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:04.341599 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" path="/var/lib/kubelet/pods/8831591d-4318-4b8d-b6bc-9790a73d8c67/volumes" Nov 01 22:48:04.342196 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:04.342180 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" path="/var/lib/kubelet/pods/b8dc27f4-6212-4492-a1d5-dcced032e3af/volumes" Nov 01 22:48:04.342673 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:04.342659 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" path="/var/lib/kubelet/pods/fdbec9fb-83d1-405b-bbcd-bc0f30257644/volumes" Nov 01 22:48:19.281003 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:48:19.280969 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:48:21.790374 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:21.790346 6466 scope.go:122] "RemoveContainer" containerID="e4bf276dd5d6b0ab312ecbd8178fa9235b8baa96bb27e1527bc1dd198a2deaec" Nov 01 22:48:49.281444 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:48:49.281405 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:48:51.436346 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:48:51.436310 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9"] Nov 01 22:49:02.619238 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:02.619134 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287"] Nov 01 22:49:06.457103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.457052 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" podUID="d4c02c43-5be8-41a0-a6e8-08ef1ee65472" containerName="main" containerID="cri-o://d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2" gracePeriod=45 Nov 01 22:49:06.725710 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.725648 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:49:06.902426 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.902398 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-model-cache\") pod \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " Nov 01 22:49:06.902620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.902440 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-kube-api-access-7q242\" (UniqueName: \"kubernetes.io/projected/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-kube-api-access-7q242\") pod \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " Nov 01 22:49:06.902620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.902469 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tmp-dir\") pod \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " Nov 01 22:49:06.902620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.902510 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-home\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-home\") pod \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " Nov 01 22:49:06.902620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.902589 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-dshm\") pod \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " Nov 01 22:49:06.902859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.902660 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tls-certs\") pod \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\" (UID: \"d4c02c43-5be8-41a0-a6e8-08ef1ee65472\") " Nov 01 22:49:06.902859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.902701 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-model-cache" pod "d4c02c43-5be8-41a0-a6e8-08ef1ee65472" (UID: "d4c02c43-5be8-41a0-a6e8-08ef1ee65472"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:06.902859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.902721 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tmp-dir" pod "d4c02c43-5be8-41a0-a6e8-08ef1ee65472" (UID: "d4c02c43-5be8-41a0-a6e8-08ef1ee65472"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:06.902859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.902796 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-home" pod "d4c02c43-5be8-41a0-a6e8-08ef1ee65472" (UID: "d4c02c43-5be8-41a0-a6e8-08ef1ee65472"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:06.903339 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.903319 6466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-home\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:06.903339 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.903341 6466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-model-cache\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:06.903497 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.903354 6466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tmp-dir\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:06.904704 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.904668 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-kube-api-access-7q242" pod "d4c02c43-5be8-41a0-a6e8-08ef1ee65472" (UID: "d4c02c43-5be8-41a0-a6e8-08ef1ee65472"). InnerVolumeSpecName "kube-api-access-7q242". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:06.904793 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.904749 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tls-certs" pod "d4c02c43-5be8-41a0-a6e8-08ef1ee65472" (UID: "d4c02c43-5be8-41a0-a6e8-08ef1ee65472"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:06.904834 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:06.904791 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-dshm" pod "d4c02c43-5be8-41a0-a6e8-08ef1ee65472" (UID: "d4c02c43-5be8-41a0-a6e8-08ef1ee65472"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:07.005045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.004968 6466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-dshm\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:07.005045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.004995 6466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-tls-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:07.005045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.005006 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q242\" (UniqueName: \"kubernetes.io/projected/d4c02c43-5be8-41a0-a6e8-08ef1ee65472-kube-api-access-7q242\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:07.035504 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.035471 6466 generic.go:356] "Generic (PLEG): container finished" podID="d4c02c43-5be8-41a0-a6e8-08ef1ee65472" containerID="d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2" exitCode=0 Nov 01 22:49:07.035681 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.035514 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" Nov 01 22:49:07.035681 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.035553 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" event={"ID":"d4c02c43-5be8-41a0-a6e8-08ef1ee65472","Type":"ContainerDied","Data":"d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2"} Nov 01 22:49:07.035681 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.035601 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9" event={"ID":"d4c02c43-5be8-41a0-a6e8-08ef1ee65472","Type":"ContainerDied","Data":"157e17419eb0c2f5e2cd9d5cf2c9fac5ffaafdf6b54d676aa8117587651fb7ee"} Nov 01 22:49:07.035681 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.035618 6466 scope.go:122] "RemoveContainer" containerID="d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2" Nov 01 22:49:07.045946 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.045929 6466 scope.go:122] "RemoveContainer" containerID="d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2" Nov 01 22:49:07.046263 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:49:07.046243 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2\": container with ID starting with d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2 not found: ID does not exist" containerID="d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2" Nov 01 22:49:07.046325 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.046272 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2"} err="failed to get container status \"d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2\": rpc error: code = NotFound desc = could not find container \"d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2\": container with ID starting with d62f50be510d251d4444d486895b9e31f4bf83f571f5e0679b919e21cb9c94a2 not found: ID does not exist" Nov 01 22:49:07.058862 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.058836 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9"] Nov 01 22:49:07.065098 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:07.065068 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1fddbd/test-model-1fddbd-kserve-597df9bcbf-zlrb9"] Nov 01 22:49:08.341044 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:08.341003 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c02c43-5be8-41a0-a6e8-08ef1ee65472" path="/var/lib/kubelet/pods/d4c02c43-5be8-41a0-a6e8-08ef1ee65472/volumes" Nov 01 22:49:12.967029 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:12.966993 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7"] Nov 01 22:49:12.994163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:12.994130 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz"] Nov 01 22:49:15.462479 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.462444 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq"] Nov 01 22:49:15.462984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.462688 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" podUID="d12d1abf-5fff-4aad-b8c1-04aa3beaa981" containerName="maas-api" containerID="cri-o://ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff" gracePeriod=30 Nov 01 22:49:15.733550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.733526 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:49:15.829920 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.829882 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-openshift-service-ca\") pod \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " Nov 01 22:49:15.830187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.830028 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-metrics-certs\") pod \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " Nov 01 22:49:15.830187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.830125 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-maas-api-tls\") pod \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " Nov 01 22:49:15.830338 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.830208 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-kube-api-access-xdfhv\" (UniqueName: \"kubernetes.io/projected/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-kube-api-access-xdfhv\") pod \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\" (UID: \"d12d1abf-5fff-4aad-b8c1-04aa3beaa981\") " Nov 01 22:49:15.837168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.837096 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-openshift-service-ca" pod "d12d1abf-5fff-4aad-b8c1-04aa3beaa981" (UID: "d12d1abf-5fff-4aad-b8c1-04aa3beaa981"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:15.852934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.852880 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-maas-api-tls" pod "d12d1abf-5fff-4aad-b8c1-04aa3beaa981" (UID: "d12d1abf-5fff-4aad-b8c1-04aa3beaa981"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:15.857682 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.857624 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-metrics-certs" pod "d12d1abf-5fff-4aad-b8c1-04aa3beaa981" (UID: "d12d1abf-5fff-4aad-b8c1-04aa3beaa981"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:15.865440 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.865324 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-kube-api-access-xdfhv" pod "d12d1abf-5fff-4aad-b8c1-04aa3beaa981" (UID: "d12d1abf-5fff-4aad-b8c1-04aa3beaa981"). InnerVolumeSpecName "kube-api-access-xdfhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:15.931724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.931671 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:15.931724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.931718 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:15.931724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.931730 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdfhv\" (UniqueName: \"kubernetes.io/projected/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-kube-api-access-xdfhv\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:15.931724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:15.931739 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12d1abf-5fff-4aad-b8c1-04aa3beaa981-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:16.091287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:16.091200 6466 generic.go:356] "Generic (PLEG): container finished" podID="d12d1abf-5fff-4aad-b8c1-04aa3beaa981" containerID="ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff" exitCode=0 Nov 01 22:49:16.091287 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:16.091245 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" Nov 01 22:49:16.091573 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:16.091291 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" event={"ID":"d12d1abf-5fff-4aad-b8c1-04aa3beaa981","Type":"ContainerDied","Data":"ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff"} Nov 01 22:49:16.091573 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:16.091354 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq" event={"ID":"d12d1abf-5fff-4aad-b8c1-04aa3beaa981","Type":"ContainerDied","Data":"de4de15ba6a0ba21d2cd342e18f4ceb203d195375f8bdd21c6f7522e8998e5b2"} Nov 01 22:49:16.091573 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:16.091382 6466 scope.go:122] "RemoveContainer" containerID="ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff" Nov 01 22:49:16.101450 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:16.101432 6466 scope.go:122] "RemoveContainer" containerID="ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff" Nov 01 22:49:16.101698 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:49:16.101678 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff\": container with ID starting with ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff not found: ID does not exist" containerID="ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff" Nov 01 22:49:16.101755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:16.101707 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff"} err="failed to get container status \"ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff\": rpc error: code = NotFound desc = could not find container \"ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff\": container with ID starting with ef02437e0e04fdf72ef035ec3e5572c901ecaa70d0d696bbcac04165623b7bff not found: ID does not exist" Nov 01 22:49:16.118363 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:16.118338 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq"] Nov 01 22:49:16.125342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:16.125322 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1fddbd-777bd466f7-887wq"] Nov 01 22:49:16.341249 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:16.341219 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12d1abf-5fff-4aad-b8c1-04aa3beaa981" path="/var/lib/kubelet/pods/d12d1abf-5fff-4aad-b8c1-04aa3beaa981/volumes" Nov 01 22:49:17.640828 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.640775 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" podUID="eb5806d0-ea66-468f-bc19-e38198c6e135" containerName="main" containerID="cri-o://a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a" gracePeriod=45 Nov 01 22:49:17.943493 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.943463 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:49:17.956080 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.956050 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb5806d0-ea66-468f-bc19-e38198c6e135-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5806d0-ea66-468f-bc19-e38198c6e135-tls-certs\") pod \"eb5806d0-ea66-468f-bc19-e38198c6e135\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " Nov 01 22:49:17.956205 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.956184 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-home\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-home\") pod \"eb5806d0-ea66-468f-bc19-e38198c6e135\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " Nov 01 22:49:17.956285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.956270 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-model-cache\") pod \"eb5806d0-ea66-468f-bc19-e38198c6e135\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " Nov 01 22:49:17.956423 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.956408 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-dshm\") pod \"eb5806d0-ea66-468f-bc19-e38198c6e135\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " Nov 01 22:49:17.956483 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.956464 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-home" pod "eb5806d0-ea66-468f-bc19-e38198c6e135" (UID: "eb5806d0-ea66-468f-bc19-e38198c6e135"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:17.956537 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.956475 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb5806d0-ea66-468f-bc19-e38198c6e135-kube-api-access-v2nwg\" (UniqueName: \"kubernetes.io/projected/eb5806d0-ea66-468f-bc19-e38198c6e135-kube-api-access-v2nwg\") pod \"eb5806d0-ea66-468f-bc19-e38198c6e135\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " Nov 01 22:49:17.956537 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.956493 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-model-cache" pod "eb5806d0-ea66-468f-bc19-e38198c6e135" (UID: "eb5806d0-ea66-468f-bc19-e38198c6e135"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:17.956537 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.956534 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-tmp-dir\") pod \"eb5806d0-ea66-468f-bc19-e38198c6e135\" (UID: \"eb5806d0-ea66-468f-bc19-e38198c6e135\") " Nov 01 22:49:17.956974 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.956947 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-tmp-dir" pod "eb5806d0-ea66-468f-bc19-e38198c6e135" (UID: "eb5806d0-ea66-468f-bc19-e38198c6e135"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:17.957144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.957121 6466 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-tmp-dir\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:17.957144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.957142 6466 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-home\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:17.957312 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.957156 6466 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-model-cache\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:17.958414 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.958383 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb5806d0-ea66-468f-bc19-e38198c6e135-tls-certs" pod "eb5806d0-ea66-468f-bc19-e38198c6e135" (UID: "eb5806d0-ea66-468f-bc19-e38198c6e135"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:17.958509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.958465 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-dshm" pod "eb5806d0-ea66-468f-bc19-e38198c6e135" (UID: "eb5806d0-ea66-468f-bc19-e38198c6e135"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:17.958816 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.958798 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5806d0-ea66-468f-bc19-e38198c6e135-kube-api-access-v2nwg" pod "eb5806d0-ea66-468f-bc19-e38198c6e135" (UID: "eb5806d0-ea66-468f-bc19-e38198c6e135"). InnerVolumeSpecName "kube-api-access-v2nwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:17.967478 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.967436 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" podUID="34b4edc8-46f2-48a0-98a8-78445a731089" containerName="payload-processing" containerID="cri-o://f8959c18ce649425f1463b08ea63f66c23e866f0747a62e90c4114f44b5c019e" gracePeriod=25 Nov 01 22:49:17.994936 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:17.994904 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" podUID="6f67dae0-ae80-4a49-81a1-b5bdab84b335" containerName="payload-pre-processing" containerID="cri-o://bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8" gracePeriod=25 Nov 01 22:49:18.058808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.058780 6466 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5806d0-ea66-468f-bc19-e38198c6e135-dshm\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:18.058808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.058807 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2nwg\" (UniqueName: \"kubernetes.io/projected/eb5806d0-ea66-468f-bc19-e38198c6e135-kube-api-access-v2nwg\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:18.059005 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.058816 6466 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5806d0-ea66-468f-bc19-e38198c6e135-tls-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:18.106163 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.106099 6466 generic.go:356] "Generic (PLEG): container finished" podID="34b4edc8-46f2-48a0-98a8-78445a731089" containerID="f8959c18ce649425f1463b08ea63f66c23e866f0747a62e90c4114f44b5c019e" exitCode=0 Nov 01 22:49:18.106319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.106180 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" event={"ID":"34b4edc8-46f2-48a0-98a8-78445a731089","Type":"ContainerDied","Data":"f8959c18ce649425f1463b08ea63f66c23e866f0747a62e90c4114f44b5c019e"} Nov 01 22:49:18.107703 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.107677 6466 generic.go:356] "Generic (PLEG): container finished" podID="eb5806d0-ea66-468f-bc19-e38198c6e135" containerID="a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a" exitCode=0 Nov 01 22:49:18.107831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.107714 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" Nov 01 22:49:18.107831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.107722 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" event={"ID":"eb5806d0-ea66-468f-bc19-e38198c6e135","Type":"ContainerDied","Data":"a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a"} Nov 01 22:49:18.107831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.107759 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287" event={"ID":"eb5806d0-ea66-468f-bc19-e38198c6e135","Type":"ContainerDied","Data":"bcdd91ceff7a46cef6854956cb8e4ac5263380684def37492fba68a564fb0537"} Nov 01 22:49:18.107831 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.107780 6466 scope.go:122] "RemoveContainer" containerID="a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a" Nov 01 22:49:18.118269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.118247 6466 scope.go:122] "RemoveContainer" containerID="a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a" Nov 01 22:49:18.118587 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:49:18.118560 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a\": container with ID starting with a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a not found: ID does not exist" containerID="a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a" Nov 01 22:49:18.118676 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.118599 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a"} err="failed to get container status \"a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a\": rpc error: code = NotFound desc = could not find container \"a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a\": container with ID starting with a22efba44a565c15e28f04dd29f19d3eecfcb773bae4d93c78e25d0d88439c2a not found: ID does not exist" Nov 01 22:49:18.137471 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.137452 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287"] Nov 01 22:49:18.154208 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.154178 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9717e2/test-model-9717e2-kserve-75bf6f7645-mb287"] Nov 01 22:49:18.221410 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.221384 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:49:18.261397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.261364 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b4edc8-46f2-48a0-98a8-78445a731089-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b4edc8-46f2-48a0-98a8-78445a731089-plugins-config-volume\") pod \"34b4edc8-46f2-48a0-98a8-78445a731089\" (UID: \"34b4edc8-46f2-48a0-98a8-78445a731089\") " Nov 01 22:49:18.261585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.261419 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b4edc8-46f2-48a0-98a8-78445a731089-kube-api-access-55kpz\" (UniqueName: \"kubernetes.io/projected/34b4edc8-46f2-48a0-98a8-78445a731089-kube-api-access-55kpz\") pod \"34b4edc8-46f2-48a0-98a8-78445a731089\" (UID: \"34b4edc8-46f2-48a0-98a8-78445a731089\") " Nov 01 22:49:18.261796 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.261767 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b4edc8-46f2-48a0-98a8-78445a731089-plugins-config-volume" pod "34b4edc8-46f2-48a0-98a8-78445a731089" (UID: "34b4edc8-46f2-48a0-98a8-78445a731089"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:18.262042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.262021 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b4edc8-46f2-48a0-98a8-78445a731089-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:18.263545 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.263521 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b4edc8-46f2-48a0-98a8-78445a731089-kube-api-access-55kpz" pod "34b4edc8-46f2-48a0-98a8-78445a731089" (UID: "34b4edc8-46f2-48a0-98a8-78445a731089"). InnerVolumeSpecName "kube-api-access-55kpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:18.341842 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.341806 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb5806d0-ea66-468f-bc19-e38198c6e135" path="/var/lib/kubelet/pods/eb5806d0-ea66-468f-bc19-e38198c6e135/volumes" Nov 01 22:49:18.362903 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.362873 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55kpz\" (UniqueName: \"kubernetes.io/projected/34b4edc8-46f2-48a0-98a8-78445a731089-kube-api-access-55kpz\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:18.560921 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:18.560834 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" podUID="6f67dae0-ae80-4a49-81a1-b5bdab84b335" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:49:19.128201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:19.125720 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" event={"ID":"34b4edc8-46f2-48a0-98a8-78445a731089","Type":"ContainerDied","Data":"1d741e209a965cd701e69c988d7bc4bca1af086163a9ce6d0d82c4e6433b862f"} Nov 01 22:49:19.128201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:19.125792 6466 scope.go:122] "RemoveContainer" containerID="f8959c18ce649425f1463b08ea63f66c23e866f0747a62e90c4114f44b5c019e" Nov 01 22:49:19.128201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:19.125902 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7" Nov 01 22:49:19.153649 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:19.153619 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7"] Nov 01 22:49:19.161647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:19.161621 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1fddbd-859c897d67-mb5m7"] Nov 01 22:49:19.358133 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:49:19.357989 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:49:20.340490 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:20.340458 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b4edc8-46f2-48a0-98a8-78445a731089" path="/var/lib/kubelet/pods/34b4edc8-46f2-48a0-98a8-78445a731089/volumes" Nov 01 22:49:24.013185 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:24.013150 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc"] Nov 01 22:49:24.013690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:24.013452 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" containerID="cri-o://58b86e24e7f8bf263c4146b6943152b17d9f8298655d035f4b39ec8b34711527" gracePeriod=30 Nov 01 22:49:24.015500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:24.015477 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd"] Nov 01 22:49:24.015676 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:24.015656 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerName="istio-proxy" containerID="cri-o://35ee1cb9d174006c8f8154cb8f22473d8b2164302221670dfe2b4eaa5736d1b9" gracePeriod=30 Nov 01 22:49:24.787245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:24.787209 6466 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:49:24.787447 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:24.787285 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:49:28.560663 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:28.560624 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" podUID="6f67dae0-ae80-4a49-81a1-b5bdab84b335" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:49:29.243415 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.243379 6466 generic.go:356] "Generic (PLEG): container finished" podID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerID="35ee1cb9d174006c8f8154cb8f22473d8b2164302221670dfe2b4eaa5736d1b9" exitCode=0 Nov 01 22:49:29.243617 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.243585 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" event={"ID":"f681a6d9-5d1e-49f5-975d-7a0c56b20329","Type":"ContainerDied","Data":"35ee1cb9d174006c8f8154cb8f22473d8b2164302221670dfe2b4eaa5736d1b9"} Nov 01 22:49:29.247527 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.247488 6466 generic.go:356] "Generic (PLEG): container finished" podID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerID="58b86e24e7f8bf263c4146b6943152b17d9f8298655d035f4b39ec8b34711527" exitCode=0 Nov 01 22:49:29.247741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.247566 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" event={"ID":"948c76bd-1965-496d-9c0d-cbb6bc453ffc","Type":"ContainerDied","Data":"58b86e24e7f8bf263c4146b6943152b17d9f8298655d035f4b39ec8b34711527"} Nov 01 22:49:29.247871 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.247860 6466 scope.go:122] "RemoveContainer" containerID="6a59eaf29dda5b74b21b0d6917afd14563d62b6b4655be6076ccb83da3b7cd4f" Nov 01 22:49:29.301155 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.301066 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:49:29.350327 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.350296 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:49:29.435234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435203 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-socket\") pod \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " Nov 01 22:49:29.435434 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435263 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-podinfo\") pod \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " Nov 01 22:49:29.435434 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435291 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-token\" (UniqueName: \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-token\") pod \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " Nov 01 22:49:29.435434 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435312 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-kube-api-access-5jzdw\" (UniqueName: \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-kube-api-access-5jzdw\") pod \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " Nov 01 22:49:29.435434 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435363 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-certs\") pod \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " Nov 01 22:49:29.435434 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435404 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-kube-api-access-5cbdd\" (UniqueName: \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-kube-api-access-5cbdd\") pod \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " Nov 01 22:49:29.435434 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435428 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-credential-socket\") pod \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " Nov 01 22:49:29.435788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435467 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-socket\") pod \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " Nov 01 22:49:29.435788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435498 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istiod-ca-cert\") pod \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " Nov 01 22:49:29.435788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435526 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-socket" pod "f681a6d9-5d1e-49f5-975d-7a0c56b20329" (UID: "f681a6d9-5d1e-49f5-975d-7a0c56b20329"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.435788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435581 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istiod-ca-cert\") pod \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " Nov 01 22:49:29.435788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435625 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-token\" (UniqueName: \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-token\") pod \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " Nov 01 22:49:29.435788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435677 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-credential-socket\") pod \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " Nov 01 22:49:29.435788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435708 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-envoy\") pod \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " Nov 01 22:49:29.435788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435741 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-data\") pod \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " Nov 01 22:49:29.435788 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435767 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-envoy\") pod \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " Nov 01 22:49:29.436272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435830 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-data\") pod \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " Nov 01 22:49:29.436272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435884 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-podinfo\") pod \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\" (UID: \"948c76bd-1965-496d-9c0d-cbb6bc453ffc\") " Nov 01 22:49:29.436272 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435919 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-certs\") pod \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\" (UID: \"f681a6d9-5d1e-49f5-975d-7a0c56b20329\") " Nov 01 22:49:29.436445 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435764 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-certs" pod "948c76bd-1965-496d-9c0d-cbb6bc453ffc" (UID: "948c76bd-1965-496d-9c0d-cbb6bc453ffc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.436445 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435804 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-credential-socket" pod "f681a6d9-5d1e-49f5-975d-7a0c56b20329" (UID: "f681a6d9-5d1e-49f5-975d-7a0c56b20329"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.436445 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.435979 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-credential-socket" pod "948c76bd-1965-496d-9c0d-cbb6bc453ffc" (UID: "948c76bd-1965-496d-9c0d-cbb6bc453ffc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.436445 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436024 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-data" pod "948c76bd-1965-496d-9c0d-cbb6bc453ffc" (UID: "948c76bd-1965-496d-9c0d-cbb6bc453ffc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.436445 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436071 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istiod-ca-cert" pod "948c76bd-1965-496d-9c0d-cbb6bc453ffc" (UID: "948c76bd-1965-496d-9c0d-cbb6bc453ffc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:29.436719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436502 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-data" pod "f681a6d9-5d1e-49f5-975d-7a0c56b20329" (UID: "f681a6d9-5d1e-49f5-975d-7a0c56b20329"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.436719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436436 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istiod-ca-cert" pod "f681a6d9-5d1e-49f5-975d-7a0c56b20329" (UID: "f681a6d9-5d1e-49f5-975d-7a0c56b20329"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:29.436719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436558 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-socket" pod "948c76bd-1965-496d-9c0d-cbb6bc453ffc" (UID: "948c76bd-1965-496d-9c0d-cbb6bc453ffc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.436719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436648 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-certs" pod "f681a6d9-5d1e-49f5-975d-7a0c56b20329" (UID: "f681a6d9-5d1e-49f5-975d-7a0c56b20329"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.436934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436851 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.436934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436876 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.436934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436891 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.436934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436906 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.436934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436920 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.436934 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436934 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.437271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436948 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.437271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436962 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.437271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436975 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.437271 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.436989 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.438559 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.438534 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-envoy" pod "f681a6d9-5d1e-49f5-975d-7a0c56b20329" (UID: "f681a6d9-5d1e-49f5-975d-7a0c56b20329"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.438743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.438550 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-token" pod "948c76bd-1965-496d-9c0d-cbb6bc453ffc" (UID: "948c76bd-1965-496d-9c0d-cbb6bc453ffc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:29.438885 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.438578 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-podinfo" pod "f681a6d9-5d1e-49f5-975d-7a0c56b20329" (UID: "f681a6d9-5d1e-49f5-975d-7a0c56b20329"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:49:29.438885 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.438602 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-kube-api-access-5jzdw" pod "948c76bd-1965-496d-9c0d-cbb6bc453ffc" (UID: "948c76bd-1965-496d-9c0d-cbb6bc453ffc"). InnerVolumeSpecName "kube-api-access-5jzdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:29.439070 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.438935 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-kube-api-access-5cbdd" pod "f681a6d9-5d1e-49f5-975d-7a0c56b20329" (UID: "f681a6d9-5d1e-49f5-975d-7a0c56b20329"). InnerVolumeSpecName "kube-api-access-5cbdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:29.439301 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.439281 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-token" pod "f681a6d9-5d1e-49f5-975d-7a0c56b20329" (UID: "f681a6d9-5d1e-49f5-975d-7a0c56b20329"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:29.439388 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.439334 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-podinfo" pod "948c76bd-1965-496d-9c0d-cbb6bc453ffc" (UID: "948c76bd-1965-496d-9c0d-cbb6bc453ffc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:49:29.439859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.439840 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-envoy" pod "948c76bd-1965-496d-9c0d-cbb6bc453ffc" (UID: "948c76bd-1965-496d-9c0d-cbb6bc453ffc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.538728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.538692 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.538728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.538719 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.538728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.538729 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.538728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.538738 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948c76bd-1965-496d-9c0d-cbb6bc453ffc-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.539090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.538747 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.539090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.538755 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.539090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.538764 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jzdw\" (UniqueName: \"kubernetes.io/projected/948c76bd-1965-496d-9c0d-cbb6bc453ffc-kube-api-access-5jzdw\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.539090 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:29.538774 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cbdd\" (UniqueName: \"kubernetes.io/projected/f681a6d9-5d1e-49f5-975d-7a0c56b20329-kube-api-access-5cbdd\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:30.254390 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:30.254360 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" Nov 01 22:49:30.254880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:30.254394 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd" event={"ID":"f681a6d9-5d1e-49f5-975d-7a0c56b20329","Type":"ContainerDied","Data":"57b398a1748e6bb162268425f059e0f44cedc86240a0e9d86759d6cb4f292f4b"} Nov 01 22:49:30.254880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:30.254462 6466 scope.go:122] "RemoveContainer" containerID="35ee1cb9d174006c8f8154cb8f22473d8b2164302221670dfe2b4eaa5736d1b9" Nov 01 22:49:30.256269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:30.256247 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" event={"ID":"948c76bd-1965-496d-9c0d-cbb6bc453ffc","Type":"ContainerDied","Data":"55f33cc262977ed6ab82fe9ab408da9b3e1ec1ef9cd94fe1f10aa6753bed1e8c"} Nov 01 22:49:30.256269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:30.256263 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc" Nov 01 22:49:30.271859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:30.271829 6466 scope.go:122] "RemoveContainer" containerID="58b86e24e7f8bf263c4146b6943152b17d9f8298655d035f4b39ec8b34711527" Nov 01 22:49:30.307393 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:30.307317 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd"] Nov 01 22:49:30.322686 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:30.322653 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-sldqd"] Nov 01 22:49:30.339688 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:30.339658 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" path="/var/lib/kubelet/pods/f681a6d9-5d1e-49f5-975d-7a0c56b20329/volumes" Nov 01 22:49:30.353592 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:30.353566 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc"] Nov 01 22:49:30.358771 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:30.358744 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-jznvc"] Nov 01 22:49:32.340134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:32.340071 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" path="/var/lib/kubelet/pods/948c76bd-1965-496d-9c0d-cbb6bc453ffc/volumes" Nov 01 22:49:34.675532 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:34.675499 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh"] Nov 01 22:49:34.690064 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:34.690029 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt"] Nov 01 22:49:36.062596 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.062562 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp"] Nov 01 22:49:36.063156 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.062827 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" podUID="ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231" containerName="maas-api" containerID="cri-o://3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c" gracePeriod=30 Nov 01 22:49:36.563121 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.563085 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:49:36.573248 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.573222 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-metrics-certs\") pod \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " Nov 01 22:49:36.573384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.573260 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-openshift-service-ca\") pod \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " Nov 01 22:49:36.573384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.573305 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-maas-api-tls\") pod \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " Nov 01 22:49:36.573384 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.573331 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-kube-api-access-2mn8z\" (UniqueName: \"kubernetes.io/projected/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-kube-api-access-2mn8z\") pod \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\" (UID: \"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231\") " Nov 01 22:49:36.573651 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.573627 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-openshift-service-ca" pod "ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231" (UID: "ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:36.573938 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.573903 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:36.575413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.575388 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-kube-api-access-2mn8z" pod "ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231" (UID: "ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231"). InnerVolumeSpecName "kube-api-access-2mn8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:36.575502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.575476 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-maas-api-tls" pod "ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231" (UID: "ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:36.575554 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.575531 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-metrics-certs" pod "ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231" (UID: "ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:36.675383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.675347 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:36.675383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.675377 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:36.675383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:36.675388 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mn8z\" (UniqueName: \"kubernetes.io/projected/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231-kube-api-access-2mn8z\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:37.298220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:37.298189 6466 generic.go:356] "Generic (PLEG): container finished" podID="ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231" containerID="3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c" exitCode=0 Nov 01 22:49:37.298647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:37.298238 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" Nov 01 22:49:37.298647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:37.298246 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" event={"ID":"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231","Type":"ContainerDied","Data":"3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c"} Nov 01 22:49:37.298647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:37.298290 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp" event={"ID":"ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231","Type":"ContainerDied","Data":"84c37d1fd3c6659798acb04a518fde734214903c8813bc15bb3407630d4779fa"} Nov 01 22:49:37.298647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:37.298310 6466 scope.go:122] "RemoveContainer" containerID="3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c" Nov 01 22:49:37.308519 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:37.308498 6466 scope.go:122] "RemoveContainer" containerID="3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c" Nov 01 22:49:37.308793 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:49:37.308771 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c\": container with ID starting with 3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c not found: ID does not exist" containerID="3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c" Nov 01 22:49:37.308852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:37.308802 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c"} err="failed to get container status \"3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c\": rpc error: code = NotFound desc = could not find container \"3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c\": container with ID starting with 3856168685d12d5f77b4dac168a1e0d0dd36c485d77eb5283ea902f9a6faf63c not found: ID does not exist" Nov 01 22:49:37.322135 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:37.322098 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp"] Nov 01 22:49:37.326804 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:37.326782 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9717e2-987f78dcb-qmdlp"] Nov 01 22:49:38.253304 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.253238 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:49:38.335473 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.335440 6466 generic.go:356] "Generic (PLEG): container finished" podID="6f67dae0-ae80-4a49-81a1-b5bdab84b335" containerID="bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8" exitCode=0 Nov 01 22:49:38.335953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.335702 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" Nov 01 22:49:38.345101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.345070 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231" path="/var/lib/kubelet/pods/ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231/volumes" Nov 01 22:49:38.347692 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.347667 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" event={"ID":"6f67dae0-ae80-4a49-81a1-b5bdab84b335","Type":"ContainerDied","Data":"bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8"} Nov 01 22:49:38.347814 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.347713 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz" event={"ID":"6f67dae0-ae80-4a49-81a1-b5bdab84b335","Type":"ContainerDied","Data":"2924be275a52e8dd363e88a143c33dbc065b941dbf466b3b8b192108a6c5a982"} Nov 01 22:49:38.347814 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.347739 6466 scope.go:122] "RemoveContainer" containerID="bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8" Nov 01 22:49:38.357496 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.357474 6466 scope.go:122] "RemoveContainer" containerID="bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8" Nov 01 22:49:38.357792 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:49:38.357773 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8\": container with ID starting with bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8 not found: ID does not exist" containerID="bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8" Nov 01 22:49:38.357840 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.357803 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8"} err="failed to get container status \"bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8\": rpc error: code = NotFound desc = could not find container \"bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8\": container with ID starting with bc2c49dd72ce7e244a17b4490e7137078708c5e1ea900831aa140fdb93b403e8 not found: ID does not exist" Nov 01 22:49:38.421041 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.421014 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f67dae0-ae80-4a49-81a1-b5bdab84b335-kube-api-access-4bqdg\" (UniqueName: \"kubernetes.io/projected/6f67dae0-ae80-4a49-81a1-b5bdab84b335-kube-api-access-4bqdg\") pod \"6f67dae0-ae80-4a49-81a1-b5bdab84b335\" (UID: \"6f67dae0-ae80-4a49-81a1-b5bdab84b335\") " Nov 01 22:49:38.421243 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.421229 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f67dae0-ae80-4a49-81a1-b5bdab84b335-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f67dae0-ae80-4a49-81a1-b5bdab84b335-plugins-config-volume\") pod \"6f67dae0-ae80-4a49-81a1-b5bdab84b335\" (UID: \"6f67dae0-ae80-4a49-81a1-b5bdab84b335\") " Nov 01 22:49:38.421680 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.421652 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f67dae0-ae80-4a49-81a1-b5bdab84b335-plugins-config-volume" pod "6f67dae0-ae80-4a49-81a1-b5bdab84b335" (UID: "6f67dae0-ae80-4a49-81a1-b5bdab84b335"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:38.423102 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.423083 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f67dae0-ae80-4a49-81a1-b5bdab84b335-kube-api-access-4bqdg" pod "6f67dae0-ae80-4a49-81a1-b5bdab84b335" (UID: "6f67dae0-ae80-4a49-81a1-b5bdab84b335"). InnerVolumeSpecName "kube-api-access-4bqdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:38.522675 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.522605 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f67dae0-ae80-4a49-81a1-b5bdab84b335-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:38.522675 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.522629 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bqdg\" (UniqueName: \"kubernetes.io/projected/6f67dae0-ae80-4a49-81a1-b5bdab84b335-kube-api-access-4bqdg\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:38.662097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.662065 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz"] Nov 01 22:49:38.664071 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:38.664047 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fddbd-7548f7864c-mq9xz"] Nov 01 22:49:39.676816 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:39.676765 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" podUID="8918ad96-0491-41aa-9a97-b67b78236470" containerName="payload-processing" containerID="cri-o://09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40" gracePeriod=25 Nov 01 22:49:39.691403 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:39.691339 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" podUID="54c5a94c-c2f6-492c-920a-efb24d34a409" containerName="payload-pre-processing" containerID="cri-o://5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c" gracePeriod=25 Nov 01 22:49:40.036941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.036914 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:49:40.040038 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.040015 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8918ad96-0491-41aa-9a97-b67b78236470-kube-api-access-x9ksr\" (UniqueName: \"kubernetes.io/projected/8918ad96-0491-41aa-9a97-b67b78236470-kube-api-access-x9ksr\") pod \"8918ad96-0491-41aa-9a97-b67b78236470\" (UID: \"8918ad96-0491-41aa-9a97-b67b78236470\") " Nov 01 22:49:40.040202 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.040187 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8918ad96-0491-41aa-9a97-b67b78236470-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8918ad96-0491-41aa-9a97-b67b78236470-plugins-config-volume\") pod \"8918ad96-0491-41aa-9a97-b67b78236470\" (UID: \"8918ad96-0491-41aa-9a97-b67b78236470\") " Nov 01 22:49:40.040589 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.040562 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8918ad96-0491-41aa-9a97-b67b78236470-plugins-config-volume" pod "8918ad96-0491-41aa-9a97-b67b78236470" (UID: "8918ad96-0491-41aa-9a97-b67b78236470"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:40.042069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.042044 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8918ad96-0491-41aa-9a97-b67b78236470-kube-api-access-x9ksr" pod "8918ad96-0491-41aa-9a97-b67b78236470" (UID: "8918ad96-0491-41aa-9a97-b67b78236470"). InnerVolumeSpecName "kube-api-access-x9ksr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:40.141981 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.141948 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8918ad96-0491-41aa-9a97-b67b78236470-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:40.141981 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.141976 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9ksr\" (UniqueName: \"kubernetes.io/projected/8918ad96-0491-41aa-9a97-b67b78236470-kube-api-access-x9ksr\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:40.339724 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.339643 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f67dae0-ae80-4a49-81a1-b5bdab84b335" path="/var/lib/kubelet/pods/6f67dae0-ae80-4a49-81a1-b5bdab84b335/volumes" Nov 01 22:49:40.352190 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.352163 6466 generic.go:356] "Generic (PLEG): container finished" podID="8918ad96-0491-41aa-9a97-b67b78236470" containerID="09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40" exitCode=0 Nov 01 22:49:40.352330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.352209 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" Nov 01 22:49:40.352330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.352241 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" event={"ID":"8918ad96-0491-41aa-9a97-b67b78236470","Type":"ContainerDied","Data":"09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40"} Nov 01 22:49:40.352330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.352295 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh" event={"ID":"8918ad96-0491-41aa-9a97-b67b78236470","Type":"ContainerDied","Data":"6bc7a586fa7d207fe2c73354c28b74d7a6bd3b180fb22155c6d6c19f224fdcef"} Nov 01 22:49:40.352330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.352325 6466 scope.go:122] "RemoveContainer" containerID="09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40" Nov 01 22:49:40.361746 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.361731 6466 scope.go:122] "RemoveContainer" containerID="09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40" Nov 01 22:49:40.362949 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:49:40.362927 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40\": container with ID starting with 09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40 not found: ID does not exist" containerID="09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40" Nov 01 22:49:40.363056 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.362954 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40"} err="failed to get container status \"09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40\": rpc error: code = NotFound desc = could not find container \"09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40\": container with ID starting with 09c14f209bcb1662d0682043a0f57b228a95de1b12de04077b101212add92d40 not found: ID does not exist" Nov 01 22:49:40.373894 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.373875 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh"] Nov 01 22:49:40.379532 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:40.379513 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9717e2-786576b5f7-z8thh"] Nov 01 22:49:42.340581 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:42.340547 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8918ad96-0491-41aa-9a97-b67b78236470" path="/var/lib/kubelet/pods/8918ad96-0491-41aa-9a97-b67b78236470/volumes" Nov 01 22:49:46.111095 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:46.111060 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt"] Nov 01 22:49:46.111520 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:46.111371 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" containerID="cri-o://824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735" gracePeriod=30 Nov 01 22:49:47.434403 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:47.434359 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" podUID="54c5a94c-c2f6-492c-920a-efb24d34a409" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:49:49.279998 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:49:49.279961 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:49:51.364877 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.364850 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:49:51.395461 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.395428 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-kube-api-access-hw7b2\" (UniqueName: \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-kube-api-access-hw7b2\") pod \"a07b617c-2d6b-4814-af51-406639a9b920\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " Nov 01 22:49:51.395641 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.395482 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-certs\") pod \"a07b617c-2d6b-4814-af51-406639a9b920\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " Nov 01 22:49:51.395641 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.395528 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a07b617c-2d6b-4814-af51-406639a9b920-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07b617c-2d6b-4814-af51-406639a9b920-istio-podinfo\") pod \"a07b617c-2d6b-4814-af51-406639a9b920\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " Nov 01 22:49:51.395641 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.395599 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-socket\") pod \"a07b617c-2d6b-4814-af51-406639a9b920\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " Nov 01 22:49:51.395833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.395681 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-istio-token\" (UniqueName: \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-istio-token\") pod \"a07b617c-2d6b-4814-af51-406639a9b920\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " Nov 01 22:49:51.395833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.395732 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-data\") pod \"a07b617c-2d6b-4814-af51-406639a9b920\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " Nov 01 22:49:51.395833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.395809 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-credential-socket\") pod \"a07b617c-2d6b-4814-af51-406639a9b920\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " Nov 01 22:49:51.396009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.395855 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07b617c-2d6b-4814-af51-406639a9b920-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07b617c-2d6b-4814-af51-406639a9b920-istiod-ca-cert\") pod \"a07b617c-2d6b-4814-af51-406639a9b920\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " Nov 01 22:49:51.396009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.395900 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-envoy\") pod \"a07b617c-2d6b-4814-af51-406639a9b920\" (UID: \"a07b617c-2d6b-4814-af51-406639a9b920\") " Nov 01 22:49:51.396009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.395913 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-socket" pod "a07b617c-2d6b-4814-af51-406639a9b920" (UID: "a07b617c-2d6b-4814-af51-406639a9b920"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:51.396330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.396091 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-data" pod "a07b617c-2d6b-4814-af51-406639a9b920" (UID: "a07b617c-2d6b-4814-af51-406639a9b920"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:51.396330 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.396159 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-credential-socket" pod "a07b617c-2d6b-4814-af51-406639a9b920" (UID: "a07b617c-2d6b-4814-af51-406639a9b920"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:51.396500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.396440 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:51.396500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.396465 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:51.396500 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.396480 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:51.396833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.396720 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07b617c-2d6b-4814-af51-406639a9b920-istiod-ca-cert" pod "a07b617c-2d6b-4814-af51-406639a9b920" (UID: "a07b617c-2d6b-4814-af51-406639a9b920"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:51.396833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.396793 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-certs" pod "a07b617c-2d6b-4814-af51-406639a9b920" (UID: "a07b617c-2d6b-4814-af51-406639a9b920"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:51.398294 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.398270 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a07b617c-2d6b-4814-af51-406639a9b920-istio-podinfo" pod "a07b617c-2d6b-4814-af51-406639a9b920" (UID: "a07b617c-2d6b-4814-af51-406639a9b920"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:49:51.398398 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.398297 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-kube-api-access-hw7b2" pod "a07b617c-2d6b-4814-af51-406639a9b920" (UID: "a07b617c-2d6b-4814-af51-406639a9b920"). InnerVolumeSpecName "kube-api-access-hw7b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:51.398507 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.398431 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-istio-token" pod "a07b617c-2d6b-4814-af51-406639a9b920" (UID: "a07b617c-2d6b-4814-af51-406639a9b920"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:51.398730 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.398712 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-envoy" pod "a07b617c-2d6b-4814-af51-406639a9b920" (UID: "a07b617c-2d6b-4814-af51-406639a9b920"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:51.436709 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.436668 6466 generic.go:356] "Generic (PLEG): container finished" podID="a07b617c-2d6b-4814-af51-406639a9b920" containerID="824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735" exitCode=0 Nov 01 22:49:51.436882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.436743 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" Nov 01 22:49:51.436882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.436753 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" event={"ID":"a07b617c-2d6b-4814-af51-406639a9b920","Type":"ContainerDied","Data":"824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735"} Nov 01 22:49:51.436882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.436811 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt" event={"ID":"a07b617c-2d6b-4814-af51-406639a9b920","Type":"ContainerDied","Data":"88b33738128128677dcb65e5e77058f3a5f7efddfe0b72807f9b23c2646cd78b"} Nov 01 22:49:51.436882 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.436836 6466 scope.go:122] "RemoveContainer" containerID="824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735" Nov 01 22:49:51.447142 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.447098 6466 scope.go:122] "RemoveContainer" containerID="13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23" Nov 01 22:49:51.455817 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.455801 6466 scope.go:122] "RemoveContainer" containerID="824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735" Nov 01 22:49:51.456071 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:49:51.456051 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735\": container with ID starting with 824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735 not found: ID does not exist" containerID="824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735" Nov 01 22:49:51.456138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.456080 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735"} err="failed to get container status \"824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735\": rpc error: code = NotFound desc = could not find container \"824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735\": container with ID starting with 824e91a54cf09458e7a1e1b3eb0be077c94f7a393593ae5195e82fe8af7a8735 not found: ID does not exist" Nov 01 22:49:51.456138 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.456097 6466 scope.go:122] "RemoveContainer" containerID="13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23" Nov 01 22:49:51.456334 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:49:51.456318 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23\": container with ID starting with 13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23 not found: ID does not exist" containerID="13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23" Nov 01 22:49:51.456380 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.456337 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23"} err="failed to get container status \"13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23\": rpc error: code = NotFound desc = could not find container \"13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23\": container with ID starting with 13af0ef81799a3907fe7d8486964daac0922dbad93cbbbb445ad7483cce61b23 not found: ID does not exist" Nov 01 22:49:51.463372 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.463344 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt"] Nov 01 22:49:51.468756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.468734 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9717e2-openshift-default-556879b8bc-xrvrt"] Nov 01 22:49:51.497481 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.497457 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07b617c-2d6b-4814-af51-406639a9b920-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:51.497481 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.497478 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:51.497605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.497490 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw7b2\" (UniqueName: \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-kube-api-access-hw7b2\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:51.497605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.497499 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07b617c-2d6b-4814-af51-406639a9b920-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:51.497605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.497508 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07b617c-2d6b-4814-af51-406639a9b920-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:51.497605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:51.497516 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07b617c-2d6b-4814-af51-406639a9b920-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:52.339859 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:52.339824 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07b617c-2d6b-4814-af51-406639a9b920" path="/var/lib/kubelet/pods/a07b617c-2d6b-4814-af51-406639a9b920/volumes" Nov 01 22:49:56.753004 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:56.752963 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75"] Nov 01 22:49:57.434342 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:57.434305 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" podUID="54c5a94c-c2f6-492c-920a-efb24d34a409" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:49:58.840036 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:58.840003 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk"] Nov 01 22:49:58.840526 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:58.840286 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" podUID="1fed40e3-422e-49b2-b12b-b92e6c53c174" containerName="maas-api" containerID="cri-o://8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91" gracePeriod=30 Nov 01 22:49:59.147743 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.147718 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:49:59.211183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.201406 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-metrics-certs\") pod \"1fed40e3-422e-49b2-b12b-b92e6c53c174\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " Nov 01 22:49:59.216229 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.215265 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fed40e3-422e-49b2-b12b-b92e6c53c174-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fed40e3-422e-49b2-b12b-b92e6c53c174-openshift-service-ca\") pod \"1fed40e3-422e-49b2-b12b-b92e6c53c174\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " Nov 01 22:49:59.216229 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.215308 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-maas-api-tls\") pod \"1fed40e3-422e-49b2-b12b-b92e6c53c174\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " Nov 01 22:49:59.216229 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.215338 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fed40e3-422e-49b2-b12b-b92e6c53c174-kube-api-access-xzgtz\" (UniqueName: \"kubernetes.io/projected/1fed40e3-422e-49b2-b12b-b92e6c53c174-kube-api-access-xzgtz\") pod \"1fed40e3-422e-49b2-b12b-b92e6c53c174\" (UID: \"1fed40e3-422e-49b2-b12b-b92e6c53c174\") " Nov 01 22:49:59.216877 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.216815 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fed40e3-422e-49b2-b12b-b92e6c53c174-openshift-service-ca" pod "1fed40e3-422e-49b2-b12b-b92e6c53c174" (UID: "1fed40e3-422e-49b2-b12b-b92e6c53c174"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:59.218888 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.218847 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fed40e3-422e-49b2-b12b-b92e6c53c174-kube-api-access-xzgtz" pod "1fed40e3-422e-49b2-b12b-b92e6c53c174" (UID: "1fed40e3-422e-49b2-b12b-b92e6c53c174"). InnerVolumeSpecName "kube-api-access-xzgtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:59.220576 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.220549 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-maas-api-tls" pod "1fed40e3-422e-49b2-b12b-b92e6c53c174" (UID: "1fed40e3-422e-49b2-b12b-b92e6c53c174"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:59.226380 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.226349 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-metrics-certs" pod "1fed40e3-422e-49b2-b12b-b92e6c53c174" (UID: "1fed40e3-422e-49b2-b12b-b92e6c53c174"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:59.317101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.317070 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fed40e3-422e-49b2-b12b-b92e6c53c174-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:59.317101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.317096 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:59.317101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.317128 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzgtz\" (UniqueName: \"kubernetes.io/projected/1fed40e3-422e-49b2-b12b-b92e6c53c174-kube-api-access-xzgtz\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:59.317351 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.317140 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fed40e3-422e-49b2-b12b-b92e6c53c174-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:59.482524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.482489 6466 generic.go:356] "Generic (PLEG): container finished" podID="1fed40e3-422e-49b2-b12b-b92e6c53c174" containerID="8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91" exitCode=0 Nov 01 22:49:59.482707 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.482537 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" Nov 01 22:49:59.482707 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.482576 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" event={"ID":"1fed40e3-422e-49b2-b12b-b92e6c53c174","Type":"ContainerDied","Data":"8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91"} Nov 01 22:49:59.482707 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.482631 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk" event={"ID":"1fed40e3-422e-49b2-b12b-b92e6c53c174","Type":"ContainerDied","Data":"0e0ca3ab01aad2e8bac7738e4cab6a24efb9fc34e6ae0049da42dda6505aecad"} Nov 01 22:49:59.482707 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.482659 6466 scope.go:122] "RemoveContainer" containerID="8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91" Nov 01 22:49:59.492296 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.492280 6466 scope.go:122] "RemoveContainer" containerID="8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91" Nov 01 22:49:59.492514 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:49:59.492497 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91\": container with ID starting with 8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91 not found: ID does not exist" containerID="8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91" Nov 01 22:49:59.492569 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.492523 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91"} err="failed to get container status \"8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91\": rpc error: code = NotFound desc = could not find container \"8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91\": container with ID starting with 8c57ede0ef94691b0b2ac8daf2e1e38ec7d59046262345e4eb087f01b6e62d91 not found: ID does not exist" Nov 01 22:49:59.523802 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.523780 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk"] Nov 01 22:49:59.528485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.528468 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93d36a-686f884f4f-bc4mk"] Nov 01 22:49:59.953665 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:49:59.953631 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:50:00.025337 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.025310 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c5a94c-c2f6-492c-920a-efb24d34a409-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c5a94c-c2f6-492c-920a-efb24d34a409-plugins-config-volume\") pod \"54c5a94c-c2f6-492c-920a-efb24d34a409\" (UID: \"54c5a94c-c2f6-492c-920a-efb24d34a409\") " Nov 01 22:50:00.025509 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.025467 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c5a94c-c2f6-492c-920a-efb24d34a409-kube-api-access-qdzzv\" (UniqueName: \"kubernetes.io/projected/54c5a94c-c2f6-492c-920a-efb24d34a409-kube-api-access-qdzzv\") pod \"54c5a94c-c2f6-492c-920a-efb24d34a409\" (UID: \"54c5a94c-c2f6-492c-920a-efb24d34a409\") " Nov 01 22:50:00.025792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.025758 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c5a94c-c2f6-492c-920a-efb24d34a409-plugins-config-volume" pod "54c5a94c-c2f6-492c-920a-efb24d34a409" (UID: "54c5a94c-c2f6-492c-920a-efb24d34a409"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:00.027606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.027579 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c5a94c-c2f6-492c-920a-efb24d34a409-kube-api-access-qdzzv" pod "54c5a94c-c2f6-492c-920a-efb24d34a409" (UID: "54c5a94c-c2f6-492c-920a-efb24d34a409"). InnerVolumeSpecName "kube-api-access-qdzzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:00.126717 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.126650 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c5a94c-c2f6-492c-920a-efb24d34a409-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:00.126717 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.126673 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdzzv\" (UniqueName: \"kubernetes.io/projected/54c5a94c-c2f6-492c-920a-efb24d34a409-kube-api-access-qdzzv\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:00.346841 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.346796 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fed40e3-422e-49b2-b12b-b92e6c53c174" path="/var/lib/kubelet/pods/1fed40e3-422e-49b2-b12b-b92e6c53c174/volumes" Nov 01 22:50:00.490696 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.490656 6466 generic.go:356] "Generic (PLEG): container finished" podID="54c5a94c-c2f6-492c-920a-efb24d34a409" containerID="5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c" exitCode=0 Nov 01 22:50:00.490884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.490721 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" Nov 01 22:50:00.490884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.490736 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" event={"ID":"54c5a94c-c2f6-492c-920a-efb24d34a409","Type":"ContainerDied","Data":"5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c"} Nov 01 22:50:00.490884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.490795 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt" event={"ID":"54c5a94c-c2f6-492c-920a-efb24d34a409","Type":"ContainerDied","Data":"38d9c124603abaa306f3561456831122a70f9b3be403a7c7183d47c3ba83751c"} Nov 01 22:50:00.490884 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.490820 6466 scope.go:122] "RemoveContainer" containerID="5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c" Nov 01 22:50:00.500026 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.500009 6466 scope.go:122] "RemoveContainer" containerID="5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c" Nov 01 22:50:00.500329 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:50:00.500307 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c\": container with ID starting with 5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c not found: ID does not exist" containerID="5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c" Nov 01 22:50:00.500423 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.500338 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c"} err="failed to get container status \"5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c\": rpc error: code = NotFound desc = could not find container \"5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c\": container with ID starting with 5f666eb0cb9d388edef2114435ec585134813cec5ff82ecac446db4ea172ce5c not found: ID does not exist" Nov 01 22:50:00.515504 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.515482 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt"] Nov 01 22:50:00.531903 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:00.531875 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9717e2-67b8b7866c-pjqtt"] Nov 01 22:50:01.753651 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:01.753582 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" podUID="d9761117-3b6c-4b40-a45d-43d23dc0c9fd" containerName="payload-pre-processing" containerID="cri-o://df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865" gracePeriod=25 Nov 01 22:50:02.057277 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.057245 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:50:02.179546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.156087 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-kube-api-access-5xvcp\" (UniqueName: \"kubernetes.io/projected/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-kube-api-access-5xvcp\") pod \"d9761117-3b6c-4b40-a45d-43d23dc0c9fd\" (UID: \"d9761117-3b6c-4b40-a45d-43d23dc0c9fd\") " Nov 01 22:50:02.179546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.156278 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-plugins-config-volume\") pod \"d9761117-3b6c-4b40-a45d-43d23dc0c9fd\" (UID: \"d9761117-3b6c-4b40-a45d-43d23dc0c9fd\") " Nov 01 22:50:02.179546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.157407 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-plugins-config-volume" pod "d9761117-3b6c-4b40-a45d-43d23dc0c9fd" (UID: "d9761117-3b6c-4b40-a45d-43d23dc0c9fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:02.179546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.160276 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-kube-api-access-5xvcp" pod "d9761117-3b6c-4b40-a45d-43d23dc0c9fd" (UID: "d9761117-3b6c-4b40-a45d-43d23dc0c9fd"). InnerVolumeSpecName "kube-api-access-5xvcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:02.257588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.257557 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:02.257588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.257585 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xvcp\" (UniqueName: \"kubernetes.io/projected/d9761117-3b6c-4b40-a45d-43d23dc0c9fd-kube-api-access-5xvcp\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:02.341699 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.341624 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c5a94c-c2f6-492c-920a-efb24d34a409" path="/var/lib/kubelet/pods/54c5a94c-c2f6-492c-920a-efb24d34a409/volumes" Nov 01 22:50:02.505846 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.505810 6466 generic.go:356] "Generic (PLEG): container finished" podID="d9761117-3b6c-4b40-a45d-43d23dc0c9fd" containerID="df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865" exitCode=0 Nov 01 22:50:02.506007 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.505858 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" Nov 01 22:50:02.506007 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.505900 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" event={"ID":"d9761117-3b6c-4b40-a45d-43d23dc0c9fd","Type":"ContainerDied","Data":"df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865"} Nov 01 22:50:02.506007 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.505962 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75" event={"ID":"d9761117-3b6c-4b40-a45d-43d23dc0c9fd","Type":"ContainerDied","Data":"646e8712673c66c846fb17040131377da8b9f4dc818400cc91f8fa944ad6031a"} Nov 01 22:50:02.506007 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.505989 6466 scope.go:122] "RemoveContainer" containerID="df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865" Nov 01 22:50:02.515858 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.515840 6466 scope.go:122] "RemoveContainer" containerID="df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865" Nov 01 22:50:02.516187 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:50:02.516167 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865\": container with ID starting with df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865 not found: ID does not exist" containerID="df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865" Nov 01 22:50:02.516269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.516200 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865"} err="failed to get container status \"df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865\": rpc error: code = NotFound desc = could not find container \"df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865\": container with ID starting with df811f04be37b375a33321758e455cda23a1085f3a34e76ada7eed1bc52a6865 not found: ID does not exist" Nov 01 22:50:02.524656 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.524623 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75"] Nov 01 22:50:02.529819 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:02.529794 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93d36a-5954f4b9ff-xmc75"] Nov 01 22:50:04.340647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:04.340614 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9761117-3b6c-4b40-a45d-43d23dc0c9fd" path="/var/lib/kubelet/pods/d9761117-3b6c-4b40-a45d-43d23dc0c9fd/volumes" Nov 01 22:50:06.722382 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:06.722348 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6"] Nov 01 22:50:06.722820 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:06.722559 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" containerID="cri-o://150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b" gracePeriod=30 Nov 01 22:50:12.400927 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.400893 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:50:12.489428 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.489378 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-kube-api-access-jwsbj\" (UniqueName: \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-kube-api-access-jwsbj\") pod \"d05a5125-b946-4a15-b8e5-663a60ffba7c\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " Nov 01 22:50:12.489750 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.489726 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-socket\") pod \"d05a5125-b946-4a15-b8e5-663a60ffba7c\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " Nov 01 22:50:12.497919 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.497670 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-socket" pod "d05a5125-b946-4a15-b8e5-663a60ffba7c" (UID: "d05a5125-b946-4a15-b8e5-663a60ffba7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:12.498311 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.498247 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-credential-socket\") pod \"d05a5125-b946-4a15-b8e5-663a60ffba7c\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " Nov 01 22:50:12.508720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.498924 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05a5125-b946-4a15-b8e5-663a60ffba7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05a5125-b946-4a15-b8e5-663a60ffba7c-istiod-ca-cert\") pod \"d05a5125-b946-4a15-b8e5-663a60ffba7c\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " Nov 01 22:50:12.508720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.498978 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-certs\") pod \"d05a5125-b946-4a15-b8e5-663a60ffba7c\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " Nov 01 22:50:12.508720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.499005 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-podinfo\") pod \"d05a5125-b946-4a15-b8e5-663a60ffba7c\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " Nov 01 22:50:12.508720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.499057 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-envoy\") pod \"d05a5125-b946-4a15-b8e5-663a60ffba7c\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " Nov 01 22:50:12.508720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.499217 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-token\" (UniqueName: \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-token\") pod \"d05a5125-b946-4a15-b8e5-663a60ffba7c\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " Nov 01 22:50:12.508720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.499271 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-data\") pod \"d05a5125-b946-4a15-b8e5-663a60ffba7c\" (UID: \"d05a5125-b946-4a15-b8e5-663a60ffba7c\") " Nov 01 22:50:12.508720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.499754 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:12.508720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.505273 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-credential-socket" pod "d05a5125-b946-4a15-b8e5-663a60ffba7c" (UID: "d05a5125-b946-4a15-b8e5-663a60ffba7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:12.508720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.506188 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-data" pod "d05a5125-b946-4a15-b8e5-663a60ffba7c" (UID: "d05a5125-b946-4a15-b8e5-663a60ffba7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:12.508720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.506928 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-certs" pod "d05a5125-b946-4a15-b8e5-663a60ffba7c" (UID: "d05a5125-b946-4a15-b8e5-663a60ffba7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:12.508720 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.507146 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05a5125-b946-4a15-b8e5-663a60ffba7c-istiod-ca-cert" pod "d05a5125-b946-4a15-b8e5-663a60ffba7c" (UID: "d05a5125-b946-4a15-b8e5-663a60ffba7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:12.521546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.521493 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-envoy" pod "d05a5125-b946-4a15-b8e5-663a60ffba7c" (UID: "d05a5125-b946-4a15-b8e5-663a60ffba7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:12.521546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.521510 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-podinfo" pod "d05a5125-b946-4a15-b8e5-663a60ffba7c" (UID: "d05a5125-b946-4a15-b8e5-663a60ffba7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:50:12.521767 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.521510 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-kube-api-access-jwsbj" pod "d05a5125-b946-4a15-b8e5-663a60ffba7c" (UID: "d05a5125-b946-4a15-b8e5-663a60ffba7c"). InnerVolumeSpecName "kube-api-access-jwsbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:12.524443 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.524402 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-token" pod "d05a5125-b946-4a15-b8e5-663a60ffba7c" (UID: "d05a5125-b946-4a15-b8e5-663a60ffba7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:12.578102 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.578068 6466 generic.go:356] "Generic (PLEG): container finished" podID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerID="150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b" exitCode=0 Nov 01 22:50:12.578102 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.578128 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" Nov 01 22:50:12.578373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.578138 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" event={"ID":"d05a5125-b946-4a15-b8e5-663a60ffba7c","Type":"ContainerDied","Data":"150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b"} Nov 01 22:50:12.578373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.578196 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6" event={"ID":"d05a5125-b946-4a15-b8e5-663a60ffba7c","Type":"ContainerDied","Data":"73dc3d6c0784bf4ff1fddd3f5c4e1f46b187f46b033c6fa6efa522b4c79c5faf"} Nov 01 22:50:12.578373 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.578220 6466 scope.go:122] "RemoveContainer" containerID="150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b" Nov 01 22:50:12.588085 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.588068 6466 scope.go:122] "RemoveContainer" containerID="7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e" Nov 01 22:50:12.597122 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.597085 6466 scope.go:122] "RemoveContainer" containerID="150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b" Nov 01 22:50:12.597433 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:50:12.597413 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b\": container with ID starting with 150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b not found: ID does not exist" containerID="150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b" Nov 01 22:50:12.597511 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.597445 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b"} err="failed to get container status \"150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b\": rpc error: code = NotFound desc = could not find container \"150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b\": container with ID starting with 150e6c9e8df8439e2eda99d3ab2878673aa03c09b9b0a6511c8580cd0624138b not found: ID does not exist" Nov 01 22:50:12.597511 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.597465 6466 scope.go:122] "RemoveContainer" containerID="7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e" Nov 01 22:50:12.597703 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:50:12.597690 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e\": container with ID starting with 7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e not found: ID does not exist" containerID="7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e" Nov 01 22:50:12.597751 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.597706 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e"} err="failed to get container status \"7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e\": rpc error: code = NotFound desc = could not find container \"7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e\": container with ID starting with 7edb3aab3b07b0009eb0fd5c054821d37ef37e4cd8b5eed7e901eff37a96578e not found: ID does not exist" Nov 01 22:50:12.601103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.600987 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:12.601103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.601018 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwsbj\" (UniqueName: \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-kube-api-access-jwsbj\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:12.601103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.601034 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:12.601103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.601046 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05a5125-b946-4a15-b8e5-663a60ffba7c-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:12.601103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.601058 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:12.601103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.601070 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:12.601103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.601081 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:12.601103 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.601092 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05a5125-b946-4a15-b8e5-663a60ffba7c-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:12.602976 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.602955 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6"] Nov 01 22:50:12.607295 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:12.607270 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93d36a-openshift-default-6945b67b9-k8rd6"] Nov 01 22:50:14.340273 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:14.340239 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" path="/var/lib/kubelet/pods/d05a5125-b946-4a15-b8e5-663a60ffba7c/volumes" Nov 01 22:50:17.419263 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:17.419224 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d"] Nov 01 22:50:17.445137 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:17.445092 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf"] Nov 01 22:50:18.031339 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.031305 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74"] Nov 01 22:50:18.031583 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.031535 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" podUID="875665f0-e7d1-49bc-af44-600e5c8e9ce1" containerName="maas-api" containerID="cri-o://e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e" gracePeriod=30 Nov 01 22:50:18.354203 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.354177 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:50:18.504063 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.504029 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/875665f0-e7d1-49bc-af44-600e5c8e9ce1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/875665f0-e7d1-49bc-af44-600e5c8e9ce1-openshift-service-ca\") pod \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " Nov 01 22:50:18.504501 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.504078 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-maas-api-tls\") pod \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " Nov 01 22:50:18.504501 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.504154 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-metrics-certs\") pod \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " Nov 01 22:50:18.504501 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.504186 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875665f0-e7d1-49bc-af44-600e5c8e9ce1-kube-api-access-7dj26\" (UniqueName: \"kubernetes.io/projected/875665f0-e7d1-49bc-af44-600e5c8e9ce1-kube-api-access-7dj26\") pod \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\" (UID: \"875665f0-e7d1-49bc-af44-600e5c8e9ce1\") " Nov 01 22:50:18.504690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.504487 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875665f0-e7d1-49bc-af44-600e5c8e9ce1-openshift-service-ca" pod "875665f0-e7d1-49bc-af44-600e5c8e9ce1" (UID: "875665f0-e7d1-49bc-af44-600e5c8e9ce1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:18.504690 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.504669 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/875665f0-e7d1-49bc-af44-600e5c8e9ce1-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:18.506424 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.506396 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-maas-api-tls" pod "875665f0-e7d1-49bc-af44-600e5c8e9ce1" (UID: "875665f0-e7d1-49bc-af44-600e5c8e9ce1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:18.506524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.506451 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-metrics-certs" pod "875665f0-e7d1-49bc-af44-600e5c8e9ce1" (UID: "875665f0-e7d1-49bc-af44-600e5c8e9ce1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:18.506524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.506476 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875665f0-e7d1-49bc-af44-600e5c8e9ce1-kube-api-access-7dj26" pod "875665f0-e7d1-49bc-af44-600e5c8e9ce1" (UID: "875665f0-e7d1-49bc-af44-600e5c8e9ce1"). InnerVolumeSpecName "kube-api-access-7dj26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:18.605957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.605871 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:18.605957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.605900 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875665f0-e7d1-49bc-af44-600e5c8e9ce1-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:18.605957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.605911 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dj26\" (UniqueName: \"kubernetes.io/projected/875665f0-e7d1-49bc-af44-600e5c8e9ce1-kube-api-access-7dj26\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:18.627491 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.627458 6466 generic.go:356] "Generic (PLEG): container finished" podID="875665f0-e7d1-49bc-af44-600e5c8e9ce1" containerID="e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e" exitCode=0 Nov 01 22:50:18.627647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.627503 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" Nov 01 22:50:18.627647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.627546 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" event={"ID":"875665f0-e7d1-49bc-af44-600e5c8e9ce1","Type":"ContainerDied","Data":"e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e"} Nov 01 22:50:18.627647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.627610 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74" event={"ID":"875665f0-e7d1-49bc-af44-600e5c8e9ce1","Type":"ContainerDied","Data":"e61cf950067879015ccd2cc5389aa4c13ae2f71ecf99757ff4859d772ffe0098"} Nov 01 22:50:18.627647 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.627635 6466 scope.go:122] "RemoveContainer" containerID="e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e" Nov 01 22:50:18.637499 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.637482 6466 scope.go:122] "RemoveContainer" containerID="e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e" Nov 01 22:50:18.637813 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:50:18.637794 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e\": container with ID starting with e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e not found: ID does not exist" containerID="e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e" Nov 01 22:50:18.637870 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.637826 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e"} err="failed to get container status \"e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e\": rpc error: code = NotFound desc = could not find container \"e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e\": container with ID starting with e3ec26470bdd148d7b68bff863bb265425b84a226344d3f918983fa7eb49177e not found: ID does not exist" Nov 01 22:50:18.651319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.651290 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74"] Nov 01 22:50:18.662174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:18.662140 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-dcd5d4-7d5dfd5658-m6f74"] Nov 01 22:50:19.280840 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:50:19.280802 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:50:20.340000 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:20.339963 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875665f0-e7d1-49bc-af44-600e5c8e9ce1" path="/var/lib/kubelet/pods/875665f0-e7d1-49bc-af44-600e5c8e9ce1/volumes" Nov 01 22:50:22.419948 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.419870 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" podUID="2b61ca43-37d4-4a9d-a4c7-0ee73742f505" containerName="payload-processing" containerID="cri-o://4d515b262943b333a1c7e4460669e5892204de5ca94377dac60c8b50868a7f51" gracePeriod=25 Nov 01 22:50:22.446264 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.446220 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" podUID="ddc44afb-2487-41d2-8883-1ae6736168b2" containerName="payload-pre-processing" containerID="cri-o://d7eb587bb0e2998fff64469867c3ff172cd6a0e3c422362efa2b0f8636ed9f33" gracePeriod=25 Nov 01 22:50:22.624544 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:50:22.624512 6466 logging.go:55] [core] [Channel #3649 SubChannel #3650]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.121:50052", ServerName: "10.134.0.121:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.121:50052: connect: connection refused" Nov 01 22:50:22.655618 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.655582 6466 generic.go:356] "Generic (PLEG): container finished" podID="2b61ca43-37d4-4a9d-a4c7-0ee73742f505" containerID="4d515b262943b333a1c7e4460669e5892204de5ca94377dac60c8b50868a7f51" exitCode=0 Nov 01 22:50:22.655783 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.655762 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" event={"ID":"2b61ca43-37d4-4a9d-a4c7-0ee73742f505","Type":"ContainerDied","Data":"4d515b262943b333a1c7e4460669e5892204de5ca94377dac60c8b50868a7f51"} Nov 01 22:50:22.657863 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.657836 6466 generic.go:356] "Generic (PLEG): container finished" podID="ddc44afb-2487-41d2-8883-1ae6736168b2" containerID="d7eb587bb0e2998fff64469867c3ff172cd6a0e3c422362efa2b0f8636ed9f33" exitCode=0 Nov 01 22:50:22.657980 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.657954 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" event={"ID":"ddc44afb-2487-41d2-8883-1ae6736168b2","Type":"ContainerDied","Data":"d7eb587bb0e2998fff64469867c3ff172cd6a0e3c422362efa2b0f8636ed9f33"} Nov 01 22:50:22.702722 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.702698 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:50:22.732727 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.732703 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:50:22.853912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.853878 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-plugins-config-volume\") pod \"2b61ca43-37d4-4a9d-a4c7-0ee73742f505\" (UID: \"2b61ca43-37d4-4a9d-a4c7-0ee73742f505\") " Nov 01 22:50:22.854086 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.853991 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc44afb-2487-41d2-8883-1ae6736168b2-kube-api-access-sqk6n\" (UniqueName: \"kubernetes.io/projected/ddc44afb-2487-41d2-8883-1ae6736168b2-kube-api-access-sqk6n\") pod \"ddc44afb-2487-41d2-8883-1ae6736168b2\" (UID: \"ddc44afb-2487-41d2-8883-1ae6736168b2\") " Nov 01 22:50:22.854180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.854097 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-kube-api-access-ckcq9\" (UniqueName: \"kubernetes.io/projected/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-kube-api-access-ckcq9\") pod \"2b61ca43-37d4-4a9d-a4c7-0ee73742f505\" (UID: \"2b61ca43-37d4-4a9d-a4c7-0ee73742f505\") " Nov 01 22:50:22.854232 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.854189 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc44afb-2487-41d2-8883-1ae6736168b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc44afb-2487-41d2-8883-1ae6736168b2-plugins-config-volume\") pod \"ddc44afb-2487-41d2-8883-1ae6736168b2\" (UID: \"ddc44afb-2487-41d2-8883-1ae6736168b2\") " Nov 01 22:50:22.854383 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.854363 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-plugins-config-volume" pod "2b61ca43-37d4-4a9d-a4c7-0ee73742f505" (UID: "2b61ca43-37d4-4a9d-a4c7-0ee73742f505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:22.854562 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.854533 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc44afb-2487-41d2-8883-1ae6736168b2-plugins-config-volume" pod "ddc44afb-2487-41d2-8883-1ae6736168b2" (UID: "ddc44afb-2487-41d2-8883-1ae6736168b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:22.854740 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.854722 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:22.854805 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.854742 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc44afb-2487-41d2-8883-1ae6736168b2-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:22.856258 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.856236 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-kube-api-access-ckcq9" pod "2b61ca43-37d4-4a9d-a4c7-0ee73742f505" (UID: "2b61ca43-37d4-4a9d-a4c7-0ee73742f505"). InnerVolumeSpecName "kube-api-access-ckcq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:22.856337 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.856259 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc44afb-2487-41d2-8883-1ae6736168b2-kube-api-access-sqk6n" pod "ddc44afb-2487-41d2-8883-1ae6736168b2" (UID: "ddc44afb-2487-41d2-8883-1ae6736168b2"). InnerVolumeSpecName "kube-api-access-sqk6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:22.956610 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.956521 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqk6n\" (UniqueName: \"kubernetes.io/projected/ddc44afb-2487-41d2-8883-1ae6736168b2-kube-api-access-sqk6n\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:22.956610 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:22.956555 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckcq9\" (UniqueName: \"kubernetes.io/projected/2b61ca43-37d4-4a9d-a4c7-0ee73742f505-kube-api-access-ckcq9\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.624201 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.624157 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" podUID="2b61ca43-37d4-4a9d-a4c7-0ee73742f505" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.120:50052\" within 1s: context deadline exceeded" Nov 01 22:50:23.624592 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:50:23.624276 6466 logging.go:55] [core] [Channel #3647 SubChannel #3648]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.120:50052", ServerName: "10.134.0.120:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.120:50052: operation was canceled" Nov 01 22:50:23.624592 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.624342 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" podUID="ddc44afb-2487-41d2-8883-1ae6736168b2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.121:50052\" within 1s: context deadline exceeded" Nov 01 22:50:23.665050 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.665023 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" Nov 01 22:50:23.665269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.665061 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d" event={"ID":"2b61ca43-37d4-4a9d-a4c7-0ee73742f505","Type":"ContainerDied","Data":"19651f9d03be34b4c04c25ea4c2b233c8a04ca551ca059bcd08c90f8427b255e"} Nov 01 22:50:23.665269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.665158 6466 scope.go:122] "RemoveContainer" containerID="4d515b262943b333a1c7e4460669e5892204de5ca94377dac60c8b50868a7f51" Nov 01 22:50:23.666528 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.666494 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" event={"ID":"ddc44afb-2487-41d2-8883-1ae6736168b2","Type":"ContainerDied","Data":"ced3ff1f5a01a9bdd4f0f311e5a977ce551b8204f5958fd95999fb53c85dded1"} Nov 01 22:50:23.666528 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.666507 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf" Nov 01 22:50:23.675704 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.675683 6466 scope.go:122] "RemoveContainer" containerID="d7eb587bb0e2998fff64469867c3ff172cd6a0e3c422362efa2b0f8636ed9f33" Nov 01 22:50:23.691411 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.691354 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d"] Nov 01 22:50:23.698096 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.698072 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dcd5d4-579fcf897c-8cn6d"] Nov 01 22:50:23.708731 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.708707 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf"] Nov 01 22:50:23.712170 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:23.712149 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-dcd5d4-65c8c5d44c-dtcmf"] Nov 01 22:50:24.339214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:24.339179 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b61ca43-37d4-4a9d-a4c7-0ee73742f505" path="/var/lib/kubelet/pods/2b61ca43-37d4-4a9d-a4c7-0ee73742f505/volumes" Nov 01 22:50:24.339692 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:24.339672 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddc44afb-2487-41d2-8883-1ae6736168b2" path="/var/lib/kubelet/pods/ddc44afb-2487-41d2-8883-1ae6736168b2/volumes" Nov 01 22:50:25.835012 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:25.834974 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9"] Nov 01 22:50:25.835515 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:25.835262 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" containerID="cri-o://06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305" gracePeriod=30 Nov 01 22:50:26.483100 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483058 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl"] Nov 01 22:50:26.483641 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483620 6466 cpu_manager.go:414] "containerMap: removing container" podUID="ddc44afb-2487-41d2-8883-1ae6736168b2" containerName="payload-pre-processing" Nov 01 22:50:26.483741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483641 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc44afb-2487-41d2-8883-1ae6736168b2" containerName="payload-pre-processing" Nov 01 22:50:26.483741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483657 6466 cpu_manager.go:414] "containerMap: removing container" podUID="54c5a94c-c2f6-492c-920a-efb24d34a409" containerName="payload-pre-processing" Nov 01 22:50:26.483741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483665 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c5a94c-c2f6-492c-920a-efb24d34a409" containerName="payload-pre-processing" Nov 01 22:50:26.483741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483676 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d12d1abf-5fff-4aad-b8c1-04aa3beaa981" containerName="maas-api" Nov 01 22:50:26.483741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483684 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12d1abf-5fff-4aad-b8c1-04aa3beaa981" containerName="maas-api" Nov 01 22:50:26.483741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483693 6466 cpu_manager.go:414] "containerMap: removing container" podUID="ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231" containerName="maas-api" Nov 01 22:50:26.483741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483699 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231" containerName="maas-api" Nov 01 22:50:26.483741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483716 6466 cpu_manager.go:414] "containerMap: removing container" podUID="1fed40e3-422e-49b2-b12b-b92e6c53c174" containerName="maas-api" Nov 01 22:50:26.483741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483723 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fed40e3-422e-49b2-b12b-b92e6c53c174" containerName="maas-api" Nov 01 22:50:26.483741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483732 6466 cpu_manager.go:414] "containerMap: removing container" podUID="2b61ca43-37d4-4a9d-a4c7-0ee73742f505" containerName="payload-processing" Nov 01 22:50:26.483741 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483739 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b61ca43-37d4-4a9d-a4c7-0ee73742f505" containerName="payload-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483750 6466 cpu_manager.go:414] "containerMap: removing container" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483757 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483775 6466 cpu_manager.go:414] "containerMap: removing container" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483783 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483796 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d4c02c43-5be8-41a0-a6e8-08ef1ee65472" containerName="main" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483803 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c02c43-5be8-41a0-a6e8-08ef1ee65472" containerName="main" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483814 6466 cpu_manager.go:414] "containerMap: removing container" podUID="eb5806d0-ea66-468f-bc19-e38198c6e135" containerName="main" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483823 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb5806d0-ea66-468f-bc19-e38198c6e135" containerName="main" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483839 6466 cpu_manager.go:414] "containerMap: removing container" podUID="34b4edc8-46f2-48a0-98a8-78445a731089" containerName="payload-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483846 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b4edc8-46f2-48a0-98a8-78445a731089" containerName="payload-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483858 6466 cpu_manager.go:414] "containerMap: removing container" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483864 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483874 6466 cpu_manager.go:414] "containerMap: removing container" podUID="875665f0-e7d1-49bc-af44-600e5c8e9ce1" containerName="maas-api" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483880 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875665f0-e7d1-49bc-af44-600e5c8e9ce1" containerName="maas-api" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483891 6466 cpu_manager.go:414] "containerMap: removing container" podUID="6f67dae0-ae80-4a49-81a1-b5bdab84b335" containerName="payload-pre-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483898 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f67dae0-ae80-4a49-81a1-b5bdab84b335" containerName="payload-pre-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483910 6466 cpu_manager.go:414] "containerMap: removing container" podUID="6c8a6a1e-70e8-4c0a-9a72-ca58a711841d" containerName="payload-pre-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483916 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8a6a1e-70e8-4c0a-9a72-ca58a711841d" containerName="payload-pre-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483930 6466 cpu_manager.go:414] "containerMap: removing container" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483938 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483955 6466 cpu_manager.go:414] "containerMap: removing container" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483962 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483971 6466 cpu_manager.go:414] "containerMap: removing container" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483978 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483991 6466 cpu_manager.go:414] "containerMap: removing container" podUID="ff764f93-2273-4694-bb85-72533fc7161c" containerName="maas-api" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.483998 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff764f93-2273-4694-bb85-72533fc7161c" containerName="maas-api" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484007 6466 cpu_manager.go:414] "containerMap: removing container" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484015 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484028 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484036 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484050 6466 cpu_manager.go:414] "containerMap: removing container" podUID="8918ad96-0491-41aa-9a97-b67b78236470" containerName="payload-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484058 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8918ad96-0491-41aa-9a97-b67b78236470" containerName="payload-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484071 6466 cpu_manager.go:414] "containerMap: removing container" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484078 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484091 6466 cpu_manager.go:414] "containerMap: removing container" podUID="2e5fd4c8-80e4-4169-a6a7-30cbee6941b8" containerName="payload-pre-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484099 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5fd4c8-80e4-4169-a6a7-30cbee6941b8" containerName="payload-pre-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484124 6466 cpu_manager.go:414] "containerMap: removing container" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484131 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484148 6466 cpu_manager.go:414] "containerMap: removing container" podUID="67394475-a855-440e-b6bd-8efb1ce7a539" containerName="payload-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484155 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67394475-a855-440e-b6bd-8efb1ce7a539" containerName="payload-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484167 6466 cpu_manager.go:414] "containerMap: removing container" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484174 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484184 6466 cpu_manager.go:414] "containerMap: removing container" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484191 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484202 6466 cpu_manager.go:414] "containerMap: removing container" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484209 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484218 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d9761117-3b6c-4b40-a45d-43d23dc0c9fd" containerName="payload-pre-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484226 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9761117-3b6c-4b40-a45d-43d23dc0c9fd" containerName="payload-pre-processing" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484237 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484243 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484257 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484264 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484277 6466 cpu_manager.go:414] "containerMap: removing container" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484284 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484297 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" Nov 01 22:50:26.484368 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484304 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484416 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c02c43-5be8-41a0-a6e8-08ef1ee65472" containerName="main" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484430 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb5806d0-ea66-468f-bc19-e38198c6e135" containerName="main" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484442 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b4edc8-46f2-48a0-98a8-78445a731089" containerName="payload-processing" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484454 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484462 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="875665f0-e7d1-49bc-af44-600e5c8e9ce1" containerName="maas-api" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484471 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f67dae0-ae80-4a49-81a1-b5bdab84b335" containerName="payload-pre-processing" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484481 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8a6a1e-70e8-4c0a-9a72-ca58a711841d" containerName="payload-pre-processing" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484491 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a87e58-fa5f-4806-81cd-bed0b1e62ece" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484499 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="024bf9b0-d2dd-4499-a45b-043af8b73da2" containerName="main" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484507 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484516 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff764f93-2273-4694-bb85-72533fc7161c" containerName="maas-api" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484526 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1daa908-9812-4ecf-923d-160fca8742c7" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484533 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="8918ad96-0491-41aa-9a97-b67b78236470" containerName="payload-processing" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484544 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484552 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5fd4c8-80e4-4169-a6a7-30cbee6941b8" containerName="payload-pre-processing" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484562 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484571 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="67394475-a855-440e-b6bd-8efb1ce7a539" containerName="payload-processing" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484582 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a12f352-2870-4eb9-992f-d1d12348f299" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484591 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbec9fb-83d1-405b-bbcd-bc0f30257644" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484599 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8dc27f4-6212-4492-a1d5-dcced032e3af" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484609 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9761117-3b6c-4b40-a45d-43d23dc0c9fd" containerName="payload-pre-processing" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484618 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f681a6d9-5d1e-49f5-975d-7a0c56b20329" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484625 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484635 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="8831591d-4318-4b8d-b6bc-9790a73d8c67" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484642 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05a5125-b946-4a15-b8e5-663a60ffba7c" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484650 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc44afb-2487-41d2-8883-1ae6736168b2" containerName="payload-pre-processing" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484659 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c5a94c-c2f6-492c-920a-efb24d34a409" containerName="payload-pre-processing" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484667 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12d1abf-5fff-4aad-b8c1-04aa3beaa981" containerName="maas-api" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484682 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef94d3c9-4aa6-40c8-9bfa-0c6f4ea35231" containerName="maas-api" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484692 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fed40e3-422e-49b2-b12b-b92e6c53c174" containerName="maas-api" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484701 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b61ca43-37d4-4a9d-a4c7-0ee73742f505" containerName="payload-processing" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484709 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="101175a2-a477-494a-99ff-ac688c53c07e" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484717 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" Nov 01 22:50:26.486728 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.484732 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="948c76bd-1965-496d-9c0d-cbb6bc453ffc" containerName="istio-proxy" Nov 01 22:50:26.489855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.489833 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:26.497305 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.496928 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:50:26.498162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.498135 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:50:26.499404 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.499376 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl"] Nov 01 22:50:26.514029 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.513995 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:50:26.514187 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.514090 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:50:26.595347 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.595304 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjzwt\" (UniqueName: \"kubernetes.io/projected/b113b598-cd51-4028-92c5-0b8d7c02f0e7-kube-api-access-bjzwt\") pod \"maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:26.595532 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.595349 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:26.595532 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.595406 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:26.696349 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.696314 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjzwt\" (UniqueName: \"kubernetes.io/projected/b113b598-cd51-4028-92c5-0b8d7c02f0e7-kube-api-access-bjzwt\") pod \"maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:26.696349 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.696356 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:26.696605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.696377 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:26.697193 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.697170 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:26.698975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.698948 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:26.706716 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.706682 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjzwt\" (UniqueName: \"kubernetes.io/projected/b113b598-cd51-4028-92c5-0b8d7c02f0e7-kube-api-access-bjzwt\") pod \"maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:26.804750 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:26.804654 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:27.080269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:27.080239 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl"] Nov 01 22:50:27.082041 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:50:27.082012 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb113b598_cd51_4028_92c5_0b8d7c02f0e7.slice/crio-cb893703f7ba14e8d0b394d06783a813b54f3cfc85430f0c7bbd20ca35c0dd47 WatchSource:0}: Error finding container cb893703f7ba14e8d0b394d06783a813b54f3cfc85430f0c7bbd20ca35c0dd47: Status 404 returned error can't find the container with id cb893703f7ba14e8d0b394d06783a813b54f3cfc85430f0c7bbd20ca35c0dd47 Nov 01 22:50:27.692562 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:27.692529 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" event={"ID":"b113b598-cd51-4028-92c5-0b8d7c02f0e7","Type":"ContainerStarted","Data":"88895663595d91dd79c812302d19e8c4f1480fae2bbbf9b35218ed212d2e6013"} Nov 01 22:50:27.692755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:27.692577 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" event={"ID":"b113b598-cd51-4028-92c5-0b8d7c02f0e7","Type":"ContainerStarted","Data":"cb893703f7ba14e8d0b394d06783a813b54f3cfc85430f0c7bbd20ca35c0dd47"} Nov 01 22:50:27.719308 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:27.719257 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" podStartSLOduration=1.719242447 podStartE2EDuration="1.719242447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:27.717644728 +0000 UTC m=+1631.502083643" watchObservedRunningTime="2026-11-01 22:50:27.719242447 +0000 UTC m=+1631.503681367" Nov 01 22:50:28.699554 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:28.699513 6466 generic.go:356] "Generic (PLEG): container finished" podID="b113b598-cd51-4028-92c5-0b8d7c02f0e7" containerID="88895663595d91dd79c812302d19e8c4f1480fae2bbbf9b35218ed212d2e6013" exitCode=0 Nov 01 22:50:28.699957 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:28.699603 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" event={"ID":"b113b598-cd51-4028-92c5-0b8d7c02f0e7","Type":"ContainerDied","Data":"88895663595d91dd79c812302d19e8c4f1480fae2bbbf9b35218ed212d2e6013"} Nov 01 22:50:29.844200 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:29.844174 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:29.934733 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:29.934703 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-openshift-ca\") pod \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " Nov 01 22:50:29.934924 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:29.934754 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-xks-ca\") pod \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " Nov 01 22:50:29.934924 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:29.934795 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b113b598-cd51-4028-92c5-0b8d7c02f0e7-kube-api-access-bjzwt\" (UniqueName: \"kubernetes.io/projected/b113b598-cd51-4028-92c5-0b8d7c02f0e7-kube-api-access-bjzwt\") pod \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\" (UID: \"b113b598-cd51-4028-92c5-0b8d7c02f0e7\") " Nov 01 22:50:29.935080 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:29.935058 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-openshift-ca" pod "b113b598-cd51-4028-92c5-0b8d7c02f0e7" (UID: "b113b598-cd51-4028-92c5-0b8d7c02f0e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:29.935211 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:29.935196 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-openshift-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:29.936930 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:29.936907 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-xks-ca" pod "b113b598-cd51-4028-92c5-0b8d7c02f0e7" (UID: "b113b598-cd51-4028-92c5-0b8d7c02f0e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:29.937065 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:29.937043 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b113b598-cd51-4028-92c5-0b8d7c02f0e7-kube-api-access-bjzwt" pod "b113b598-cd51-4028-92c5-0b8d7c02f0e7" (UID: "b113b598-cd51-4028-92c5-0b8d7c02f0e7"). InnerVolumeSpecName "kube-api-access-bjzwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:30.037033 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:30.036944 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b113b598-cd51-4028-92c5-0b8d7c02f0e7-maas-api-xks-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:30.037033 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:30.036976 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjzwt\" (UniqueName: \"kubernetes.io/projected/b113b598-cd51-4028-92c5-0b8d7c02f0e7-kube-api-access-bjzwt\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:30.713792 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:30.713758 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" event={"ID":"b113b598-cd51-4028-92c5-0b8d7c02f0e7","Type":"ContainerDied","Data":"cb893703f7ba14e8d0b394d06783a813b54f3cfc85430f0c7bbd20ca35c0dd47"} Nov 01 22:50:30.713947 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:30.713807 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb893703f7ba14e8d0b394d06783a813b54f3cfc85430f0c7bbd20ca35c0dd47" Nov 01 22:50:30.713947 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:30.713770 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl" Nov 01 22:50:31.117269 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.117244 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:50:31.251546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.251515 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istiod-ca-cert\") pod \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " Nov 01 22:50:31.251546 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.251554 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-envoy\") pod \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " Nov 01 22:50:31.251835 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.251621 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-data\") pod \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " Nov 01 22:50:31.251835 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.251733 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-certs\") pod \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " Nov 01 22:50:31.251835 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.251781 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-podinfo\") pod \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " Nov 01 22:50:31.252002 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.251847 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-credential-socket\") pod \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " Nov 01 22:50:31.252002 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.251888 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-data" pod "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" (UID: "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.252002 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.251920 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-kube-api-access-t4z6x\" (UniqueName: \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-kube-api-access-t4z6x\") pod \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " Nov 01 22:50:31.252002 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.251952 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-socket\") pod \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " Nov 01 22:50:31.252002 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.251981 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-token\" (UniqueName: \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-token\") pod \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\" (UID: \"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1\") " Nov 01 22:50:31.252297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.252141 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-credential-socket" pod "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" (UID: "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.252297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.252218 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-certs" pod "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" (UID: "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.252475 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.252342 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-socket" pod "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" (UID: "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.252578 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.252555 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.252638 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.252614 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.252638 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.252630 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.252756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.252643 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.252816 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.252763 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istiod-ca-cert" pod "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" (UID: "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:31.254574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.254477 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-kube-api-access-t4z6x" pod "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" (UID: "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1"). InnerVolumeSpecName "kube-api-access-t4z6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:31.254800 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.254776 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-podinfo" pod "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" (UID: "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:50:31.254953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.254927 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-envoy" pod "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" (UID: "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.255045 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.254979 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-token" pod "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" (UID: "df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:31.354278 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.354242 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.354278 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.354270 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.354278 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.354280 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.354530 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.354290 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4z6x\" (UniqueName: \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-kube-api-access-t4z6x\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.354530 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.354300 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.720693 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.720650 6466 generic.go:356] "Generic (PLEG): container finished" podID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerID="06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305" exitCode=0 Nov 01 22:50:31.720880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.720716 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" Nov 01 22:50:31.720880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.720732 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" event={"ID":"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1","Type":"ContainerDied","Data":"06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305"} Nov 01 22:50:31.720880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.720786 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9" event={"ID":"df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1","Type":"ContainerDied","Data":"ed87d02f50561f507f0060a89e6d6c2d691306b70e720be117e2cf153293ea26"} Nov 01 22:50:31.720880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.720808 6466 scope.go:122] "RemoveContainer" containerID="06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305" Nov 01 22:50:31.731851 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.731830 6466 scope.go:122] "RemoveContainer" containerID="06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305" Nov 01 22:50:31.732161 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:50:31.732141 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305\": container with ID starting with 06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305 not found: ID does not exist" containerID="06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305" Nov 01 22:50:31.732233 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.732171 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305"} err="failed to get container status \"06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305\": rpc error: code = NotFound desc = could not find container \"06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305\": container with ID starting with 06c3deefce23b86d2ea085a8d7f90ac6ac63602a1e3573aa8f03c3bbeb269305 not found: ID does not exist" Nov 01 22:50:31.744248 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.744217 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9"] Nov 01 22:50:31.750297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:31.750267 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-mwxs9"] Nov 01 22:50:32.340130 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:32.340069 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" path="/var/lib/kubelet/pods/df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1/volumes" Nov 01 22:50:37.013853 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.013820 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn"] Nov 01 22:50:37.014292 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014279 6466 cpu_manager.go:414] "containerMap: removing container" podUID="b113b598-cd51-4028-92c5-0b8d7c02f0e7" containerName="revoke-keys" Nov 01 22:50:37.014334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014293 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b113b598-cd51-4028-92c5-0b8d7c02f0e7" containerName="revoke-keys" Nov 01 22:50:37.014334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014305 6466 cpu_manager.go:414] "containerMap: removing container" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" Nov 01 22:50:37.014334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014310 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" Nov 01 22:50:37.014334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014320 6466 cpu_manager.go:414] "containerMap: removing container" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" Nov 01 22:50:37.014334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014325 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" Nov 01 22:50:37.014334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014333 6466 cpu_manager.go:414] "containerMap: removing container" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" Nov 01 22:50:37.014524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014340 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" Nov 01 22:50:37.014524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014407 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b113b598-cd51-4028-92c5-0b8d7c02f0e7" containerName="revoke-keys" Nov 01 22:50:37.014524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014416 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="53147cfc-27a6-4f61-863a-732bfd063831" containerName="istio-proxy" Nov 01 22:50:37.014524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014422 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="df23a08b-7aa1-453c-afb4-f9a7a9c9e9d1" containerName="istio-proxy" Nov 01 22:50:37.014524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.014428 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07b617c-2d6b-4814-af51-406639a9b920" containerName="istio-proxy" Nov 01 22:50:37.017605 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.017582 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:37.021805 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.021780 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:50:37.025551 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.025529 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:50:37.026697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.026664 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0fa409-65kmn\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:37.026833 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.026815 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns9pq\" (UniqueName: \"kubernetes.io/projected/41d9aead-f494-428f-954a-9f4544bdb67d-kube-api-access-ns9pq\") pod \"maas-api-revoke-sub-6a0fa409-65kmn\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:37.026968 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.026943 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0fa409-65kmn\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:37.027043 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.026998 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:50:37.027175 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.027153 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:50:37.028796 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.028770 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn"] Nov 01 22:50:37.127991 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.127953 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0fa409-65kmn\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:37.128252 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.128042 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns9pq\" (UniqueName: \"kubernetes.io/projected/41d9aead-f494-428f-954a-9f4544bdb67d-kube-api-access-ns9pq\") pod \"maas-api-revoke-sub-6a0fa409-65kmn\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:37.128252 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.128073 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0fa409-65kmn\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:37.128706 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.128670 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a0fa409-65kmn\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:37.130435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.130415 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a0fa409-65kmn\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:37.136319 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.136297 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns9pq\" (UniqueName: \"kubernetes.io/projected/41d9aead-f494-428f-954a-9f4544bdb67d-kube-api-access-ns9pq\") pod \"maas-api-revoke-sub-6a0fa409-65kmn\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:37.332585 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.332499 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:37.608924 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.608897 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn"] Nov 01 22:50:37.611029 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:50:37.611006 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d9aead_f494_428f_954a_9f4544bdb67d.slice/crio-e2c6202b54db5f3841117ec0f91dcdb3080c090adee0db9dcdf12425088cbca9 WatchSource:0}: Error finding container e2c6202b54db5f3841117ec0f91dcdb3080c090adee0db9dcdf12425088cbca9: Status 404 returned error can't find the container with id e2c6202b54db5f3841117ec0f91dcdb3080c090adee0db9dcdf12425088cbca9 Nov 01 22:50:37.761935 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:37.761896 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" event={"ID":"41d9aead-f494-428f-954a-9f4544bdb67d","Type":"ContainerStarted","Data":"e2c6202b54db5f3841117ec0f91dcdb3080c090adee0db9dcdf12425088cbca9"} Nov 01 22:50:38.770561 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:38.770526 6466 generic.go:356] "Generic (PLEG): container finished" podID="41d9aead-f494-428f-954a-9f4544bdb67d" containerID="2ad302274dc5b1d73fc602d1dbbf7b9ad8215c8a8b8a5d224abb8e81aa26236a" exitCode=0 Nov 01 22:50:38.770954 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:38.770609 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" event={"ID":"41d9aead-f494-428f-954a-9f4544bdb67d","Type":"ContainerDied","Data":"2ad302274dc5b1d73fc602d1dbbf7b9ad8215c8a8b8a5d224abb8e81aa26236a"} Nov 01 22:50:39.932456 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:39.932430 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:39.960212 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:39.960178 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-openshift-ca\") pod \"41d9aead-f494-428f-954a-9f4544bdb67d\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " Nov 01 22:50:39.960361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:39.960285 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d9aead-f494-428f-954a-9f4544bdb67d-kube-api-access-ns9pq\" (UniqueName: \"kubernetes.io/projected/41d9aead-f494-428f-954a-9f4544bdb67d-kube-api-access-ns9pq\") pod \"41d9aead-f494-428f-954a-9f4544bdb67d\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " Nov 01 22:50:39.960417 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:39.960384 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-xks-ca\") pod \"41d9aead-f494-428f-954a-9f4544bdb67d\" (UID: \"41d9aead-f494-428f-954a-9f4544bdb67d\") " Nov 01 22:50:39.960619 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:39.960596 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-openshift-ca" pod "41d9aead-f494-428f-954a-9f4544bdb67d" (UID: "41d9aead-f494-428f-954a-9f4544bdb67d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:39.960808 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:39.960789 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-openshift-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:39.962474 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:39.962446 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-xks-ca" pod "41d9aead-f494-428f-954a-9f4544bdb67d" (UID: "41d9aead-f494-428f-954a-9f4544bdb67d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:39.962586 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:39.962470 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d9aead-f494-428f-954a-9f4544bdb67d-kube-api-access-ns9pq" pod "41d9aead-f494-428f-954a-9f4544bdb67d" (UID: "41d9aead-f494-428f-954a-9f4544bdb67d"). InnerVolumeSpecName "kube-api-access-ns9pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:40.061455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:40.061371 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns9pq\" (UniqueName: \"kubernetes.io/projected/41d9aead-f494-428f-954a-9f4544bdb67d-kube-api-access-ns9pq\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:40.061455 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:40.061400 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d9aead-f494-428f-954a-9f4544bdb67d-maas-api-xks-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:40.781738 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:40.781709 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" Nov 01 22:50:40.781912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:40.781743 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn" event={"ID":"41d9aead-f494-428f-954a-9f4544bdb67d","Type":"ContainerDied","Data":"e2c6202b54db5f3841117ec0f91dcdb3080c090adee0db9dcdf12425088cbca9"} Nov 01 22:50:40.781912 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:40.781797 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2c6202b54db5f3841117ec0f91dcdb3080c090adee0db9dcdf12425088cbca9" Nov 01 22:50:49.302480 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:50:49.302375 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:50:52.395464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:52.395427 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl"] Nov 01 22:50:52.401679 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:52.401653 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-aa441b-w26gl"] Nov 01 22:50:54.352805 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:54.352766 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b113b598-cd51-4028-92c5-0b8d7c02f0e7" path="/var/lib/kubelet/pods/b113b598-cd51-4028-92c5-0b8d7c02f0e7/volumes" Nov 01 22:50:59.520762 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:59.520732 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb"] Nov 01 22:50:59.521172 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:59.520929 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" containerID="cri-o://2f04610032a78ebc8a14ee695f03157874794ad482483e02c8e6e4477f5650c7" gracePeriod=30 Nov 01 22:50:59.522408 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:59.522377 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n"] Nov 01 22:50:59.522653 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:50:59.522612 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" containerID="cri-o://5cfe1758824da5e20ec321751dca2e6669576c740560ac37b024e511267b968b" gracePeriod=30 Nov 01 22:51:00.296620 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:00.296584 6466 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:51:00.296846 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:00.296667 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:51:03.472351 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.472308 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b57d74f86-drs46"] Nov 01 22:51:03.473132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.473090 6466 cpu_manager.go:414] "containerMap: removing container" podUID="41d9aead-f494-428f-954a-9f4544bdb67d" containerName="revoke-keys" Nov 01 22:51:03.473262 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.473133 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d9aead-f494-428f-954a-9f4544bdb67d" containerName="revoke-keys" Nov 01 22:51:03.473262 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.473247 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d9aead-f494-428f-954a-9f4544bdb67d" containerName="revoke-keys" Nov 01 22:51:03.478361 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.478331 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:03.490521 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.489820 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 22:51:03.490521 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.489989 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-m2mcq\"" Nov 01 22:51:03.490521 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.490100 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 22:51:03.490521 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.490222 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 22:51:03.496975 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.496945 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b57d74f86-drs46"] Nov 01 22:51:03.526809 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.526761 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-metrics-certs\") pod \"maas-controller-7b57d74f86-drs46\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:03.527082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.526829 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-webhook-cert\") pod \"maas-controller-7b57d74f86-drs46\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:03.527082 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.526867 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sk7t\" (UniqueName: \"kubernetes.io/projected/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-kube-api-access-5sk7t\") pod \"maas-controller-7b57d74f86-drs46\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:03.627783 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.627731 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-metrics-certs\") pod \"maas-controller-7b57d74f86-drs46\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:03.628001 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.627799 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-webhook-cert\") pod \"maas-controller-7b57d74f86-drs46\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:03.628001 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.627834 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sk7t\" (UniqueName: \"kubernetes.io/projected/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-kube-api-access-5sk7t\") pod \"maas-controller-7b57d74f86-drs46\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:03.631334 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.631301 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-webhook-cert\") pod \"maas-controller-7b57d74f86-drs46\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:03.631893 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.631868 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-metrics-certs\") pod \"maas-controller-7b57d74f86-drs46\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:03.642170 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.641870 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sk7t\" (UniqueName: \"kubernetes.io/projected/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-kube-api-access-5sk7t\") pod \"maas-controller-7b57d74f86-drs46\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:03.800009 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:03.799899 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:04.117218 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:04.116262 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b57d74f86-drs46"] Nov 01 22:51:04.148144 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:51:04.147869 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4b1b0b_6012_42e7_b4d1_2a7293549ea0.slice/crio-68c9582b015706a47cfbdfd6927ab9f6955195212b1d0dcae06a12100f5f1d8a WatchSource:0}: Error finding container 68c9582b015706a47cfbdfd6927ab9f6955195212b1d0dcae06a12100f5f1d8a: Status 404 returned error can't find the container with id 68c9582b015706a47cfbdfd6927ab9f6955195212b1d0dcae06a12100f5f1d8a Nov 01 22:51:04.176325 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:04.160342 6466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:51:04.941132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:04.941061 6466 generic.go:356] "Generic (PLEG): container finished" podID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerID="2f04610032a78ebc8a14ee695f03157874794ad482483e02c8e6e4477f5650c7" exitCode=0 Nov 01 22:51:04.941667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:04.941229 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" event={"ID":"af543f1a-d698-4a5c-bf14-49e030f8b80e","Type":"ContainerDied","Data":"2f04610032a78ebc8a14ee695f03157874794ad482483e02c8e6e4477f5650c7"} Nov 01 22:51:04.957134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:04.954664 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b57d74f86-drs46" event={"ID":"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0","Type":"ContainerStarted","Data":"68c9582b015706a47cfbdfd6927ab9f6955195212b1d0dcae06a12100f5f1d8a"} Nov 01 22:51:04.969134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:04.966646 6466 generic.go:356] "Generic (PLEG): container finished" podID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerID="5cfe1758824da5e20ec321751dca2e6669576c740560ac37b024e511267b968b" exitCode=0 Nov 01 22:51:04.969134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:04.966708 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" event={"ID":"b00c4715-e3be-4ea0-8ba6-6d78396fdf18","Type":"ContainerDied","Data":"5cfe1758824da5e20ec321751dca2e6669576c740560ac37b024e511267b968b"} Nov 01 22:51:05.642307 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.642276 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:51:05.731841 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.731800 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-socket\") pod \"af543f1a-d698-4a5c-bf14-49e030f8b80e\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " Nov 01 22:51:05.732157 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.732101 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-envoy\") pod \"af543f1a-d698-4a5c-bf14-49e030f8b80e\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " Nov 01 22:51:05.732397 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.732380 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-certs\") pod \"af543f1a-d698-4a5c-bf14-49e030f8b80e\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " Nov 01 22:51:05.732539 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.732525 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-credential-socket\") pod \"af543f1a-d698-4a5c-bf14-49e030f8b80e\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " Nov 01 22:51:05.732667 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.732650 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-token\" (UniqueName: \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-token\") pod \"af543f1a-d698-4a5c-bf14-49e030f8b80e\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " Nov 01 22:51:05.732799 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.732785 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-podinfo\") pod \"af543f1a-d698-4a5c-bf14-49e030f8b80e\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " Nov 01 22:51:05.732931 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.732918 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-data\") pod \"af543f1a-d698-4a5c-bf14-49e030f8b80e\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " Nov 01 22:51:05.733071 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.733055 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af543f1a-d698-4a5c-bf14-49e030f8b80e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af543f1a-d698-4a5c-bf14-49e030f8b80e-istiod-ca-cert\") pod \"af543f1a-d698-4a5c-bf14-49e030f8b80e\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " Nov 01 22:51:05.733291 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.733273 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-kube-api-access-fbxmn\" (UniqueName: \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-kube-api-access-fbxmn\") pod \"af543f1a-d698-4a5c-bf14-49e030f8b80e\" (UID: \"af543f1a-d698-4a5c-bf14-49e030f8b80e\") " Nov 01 22:51:05.734435 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.734249 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-credential-socket" pod "af543f1a-d698-4a5c-bf14-49e030f8b80e" (UID: "af543f1a-d698-4a5c-bf14-49e030f8b80e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.734739 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.734574 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-certs" pod "af543f1a-d698-4a5c-bf14-49e030f8b80e" (UID: "af543f1a-d698-4a5c-bf14-49e030f8b80e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.735012 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.734857 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-socket" pod "af543f1a-d698-4a5c-bf14-49e030f8b80e" (UID: "af543f1a-d698-4a5c-bf14-49e030f8b80e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.747237 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.747187 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-data" pod "af543f1a-d698-4a5c-bf14-49e030f8b80e" (UID: "af543f1a-d698-4a5c-bf14-49e030f8b80e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.825160 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.758072 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af543f1a-d698-4a5c-bf14-49e030f8b80e-istiod-ca-cert" pod "af543f1a-d698-4a5c-bf14-49e030f8b80e" (UID: "af543f1a-d698-4a5c-bf14-49e030f8b80e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:51:05.837463 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.837434 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.841275 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.841241 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.841446 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.841431 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af543f1a-d698-4a5c-bf14-49e030f8b80e-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.841588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.841574 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.841752 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.841705 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.841752 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.838610 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-token" pod "af543f1a-d698-4a5c-bf14-49e030f8b80e" (UID: "af543f1a-d698-4a5c-bf14-49e030f8b80e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:05.841752 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.838950 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-envoy" pod "af543f1a-d698-4a5c-bf14-49e030f8b80e" (UID: "af543f1a-d698-4a5c-bf14-49e030f8b80e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.895498 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.895380 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-podinfo" pod "af543f1a-d698-4a5c-bf14-49e030f8b80e" (UID: "af543f1a-d698-4a5c-bf14-49e030f8b80e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:51:05.920169 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.917073 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-kube-api-access-fbxmn" pod "af543f1a-d698-4a5c-bf14-49e030f8b80e" (UID: "af543f1a-d698-4a5c-bf14-49e030f8b80e"). InnerVolumeSpecName "kube-api-access-fbxmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:05.924143 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.921450 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945079 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-data\") pod \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945178 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-socket\") pod \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945234 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-kube-api-access-hlnhh\" (UniqueName: \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-kube-api-access-hlnhh\") pod \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945319 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-envoy\") pod \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945365 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-podinfo\") pod \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945397 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istiod-ca-cert\") pod \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945425 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-token\" (UniqueName: \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-token\") pod \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945462 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-credential-socket\") pod \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945489 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-certs\") pod \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\" (UID: \"b00c4715-e3be-4ea0-8ba6-6d78396fdf18\") " Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945821 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945839 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945853 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbxmn\" (UniqueName: \"kubernetes.io/projected/af543f1a-d698-4a5c-bf14-49e030f8b80e-kube-api-access-fbxmn\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.945867 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af543f1a-d698-4a5c-bf14-49e030f8b80e-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.961221 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-certs" pod "b00c4715-e3be-4ea0-8ba6-6d78396fdf18" (UID: "b00c4715-e3be-4ea0-8ba6-6d78396fdf18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.966212 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-data" pod "b00c4715-e3be-4ea0-8ba6-6d78396fdf18" (UID: "b00c4715-e3be-4ea0-8ba6-6d78396fdf18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:05.966484 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-socket" pod "b00c4715-e3be-4ea0-8ba6-6d78396fdf18" (UID: "b00c4715-e3be-4ea0-8ba6-6d78396fdf18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:06.005457 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.005048 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istiod-ca-cert" pod "b00c4715-e3be-4ea0-8ba6-6d78396fdf18" (UID: "b00c4715-e3be-4ea0-8ba6-6d78396fdf18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:51:06.023132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.009192 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-credential-socket" pod "b00c4715-e3be-4ea0-8ba6-6d78396fdf18" (UID: "b00c4715-e3be-4ea0-8ba6-6d78396fdf18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:06.023132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.012280 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-kube-api-access-hlnhh" pod "b00c4715-e3be-4ea0-8ba6-6d78396fdf18" (UID: "b00c4715-e3be-4ea0-8ba6-6d78396fdf18"). InnerVolumeSpecName "kube-api-access-hlnhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:06.023132 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.012325 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-token" pod "b00c4715-e3be-4ea0-8ba6-6d78396fdf18" (UID: "b00c4715-e3be-4ea0-8ba6-6d78396fdf18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:06.057134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.056677 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.057134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.056724 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.057134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.056745 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlnhh\" (UniqueName: \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-kube-api-access-hlnhh\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.057134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.056762 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.057134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.056777 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.057134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.056793 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.057134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.056807 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.080134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.057729 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" event={"ID":"af543f1a-d698-4a5c-bf14-49e030f8b80e","Type":"ContainerDied","Data":"0197542d9a7e8af418f4558d65d1fd5b8fbd790694b5b147e15d73a971811198"} Nov 01 22:51:06.080134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.057803 6466 scope.go:122] "RemoveContainer" containerID="2f04610032a78ebc8a14ee695f03157874794ad482483e02c8e6e4477f5650c7" Nov 01 22:51:06.080134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.057973 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb" Nov 01 22:51:06.089333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.084612 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" event={"ID":"b00c4715-e3be-4ea0-8ba6-6d78396fdf18","Type":"ContainerDied","Data":"daa17676493be99ce254c5b267b265e52ead8462988c684230b640490acf1a99"} Nov 01 22:51:06.089333 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.084743 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n" Nov 01 22:51:06.214489 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.211312 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-envoy" pod "b00c4715-e3be-4ea0-8ba6-6d78396fdf18" (UID: "b00c4715-e3be-4ea0-8ba6-6d78396fdf18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:06.226316 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.225274 6466 scope.go:122] "RemoveContainer" containerID="5cfe1758824da5e20ec321751dca2e6669576c740560ac37b024e511267b968b" Nov 01 22:51:06.226971 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.226893 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-podinfo" pod "b00c4715-e3be-4ea0-8ba6-6d78396fdf18" (UID: "b00c4715-e3be-4ea0-8ba6-6d78396fdf18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:51:06.262955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.262802 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.262955 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.262835 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00c4715-e3be-4ea0-8ba6-6d78396fdf18-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:06.302516 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.301280 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb"] Nov 01 22:51:06.304951 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.304924 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-nn2sb"] Nov 01 22:51:06.346050 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.346004 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" path="/var/lib/kubelet/pods/af543f1a-d698-4a5c-bf14-49e030f8b80e/volumes" Nov 01 22:51:06.402329 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.402294 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n"] Nov 01 22:51:06.405683 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:06.405656 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-qz85n"] Nov 01 22:51:08.342472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:08.342434 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" path="/var/lib/kubelet/pods/b00c4715-e3be-4ea0-8ba6-6d78396fdf18/volumes" Nov 01 22:51:09.104881 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:09.104841 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b57d74f86-drs46" event={"ID":"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0","Type":"ContainerStarted","Data":"dee6c5312a0f16ff373d880fe537bfc97f00a167899b1601a158aff9f13eb193"} Nov 01 22:51:09.105089 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:09.104910 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:09.122860 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:09.122762 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b57d74f86-drs46" podStartSLOduration=1.347127508 podStartE2EDuration="6.122747288s" totalImagesPullingTime="4.77561978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:09.122625934 +0000 UTC m=+1672.907064849" watchObservedRunningTime="2026-11-01 22:51:09.122747288 +0000 UTC m=+1672.907186209" Nov 01 22:51:19.280127 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:51:19.280061 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:51:20.114761 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:20.114733 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:25.474343 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.474311 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf"] Nov 01 22:51:25.474755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.474734 6466 cpu_manager.go:414] "containerMap: removing container" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" Nov 01 22:51:25.474755 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.474746 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" Nov 01 22:51:25.474832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.474765 6466 cpu_manager.go:414] "containerMap: removing container" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" Nov 01 22:51:25.474832 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.474770 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" Nov 01 22:51:25.474896 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.474845 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="af543f1a-d698-4a5c-bf14-49e030f8b80e" containerName="istio-proxy" Nov 01 22:51:25.474896 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.474861 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00c4715-e3be-4ea0-8ba6-6d78396fdf18" containerName="istio-proxy" Nov 01 22:51:25.479698 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.479677 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:25.485180 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.485152 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:51:25.485332 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.485273 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:51:25.487072 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.487048 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf"] Nov 01 22:51:25.495544 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.495522 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:51:25.495666 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.495602 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:51:25.552787 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.552745 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mkcjf\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:25.552959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.552805 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mkcjf\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:25.553026 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.553006 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95jv6\" (UniqueName: \"kubernetes.io/projected/846d6de0-2050-42ef-beef-2a72969adc5a-kube-api-access-95jv6\") pod \"maas-api-revoke-sub-85237e45-mkcjf\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:25.654049 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.654003 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mkcjf\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:25.654049 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.654056 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mkcjf\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:25.654349 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.654197 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95jv6\" (UniqueName: \"kubernetes.io/projected/846d6de0-2050-42ef-beef-2a72969adc5a-kube-api-access-95jv6\") pod \"maas-api-revoke-sub-85237e45-mkcjf\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:25.654941 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.654902 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mkcjf\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:25.656697 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.656666 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mkcjf\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:25.662523 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.662502 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95jv6\" (UniqueName: \"kubernetes.io/projected/846d6de0-2050-42ef-beef-2a72969adc5a-kube-api-access-95jv6\") pod \"maas-api-revoke-sub-85237e45-mkcjf\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:25.797785 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:25.797694 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:26.091152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:26.091120 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf"] Nov 01 22:51:26.093845 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:51:26.093812 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846d6de0_2050_42ef_beef_2a72969adc5a.slice/crio-16eda4a7a0a5790368d46a8826b23c1178d2488bd05426f6f6960f387cfb3ef5 WatchSource:0}: Error finding container 16eda4a7a0a5790368d46a8826b23c1178d2488bd05426f6f6960f387cfb3ef5: Status 404 returned error can't find the container with id 16eda4a7a0a5790368d46a8826b23c1178d2488bd05426f6f6960f387cfb3ef5 Nov 01 22:51:26.193392 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:26.193354 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" event={"ID":"846d6de0-2050-42ef-beef-2a72969adc5a","Type":"ContainerStarted","Data":"16eda4a7a0a5790368d46a8826b23c1178d2488bd05426f6f6960f387cfb3ef5"} Nov 01 22:51:27.201037 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:27.200995 6466 generic.go:356] "Generic (PLEG): container finished" podID="846d6de0-2050-42ef-beef-2a72969adc5a" containerID="b6e9d9e014d72e72e0eba0e32bd4967ddc6eb41b42505cd8dab1555a6c3d6f26" exitCode=0 Nov 01 22:51:27.201574 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:27.201176 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" event={"ID":"846d6de0-2050-42ef-beef-2a72969adc5a","Type":"ContainerDied","Data":"b6e9d9e014d72e72e0eba0e32bd4967ddc6eb41b42505cd8dab1555a6c3d6f26"} Nov 01 22:51:28.349903 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:28.349873 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:28.485427 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:28.485349 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846d6de0-2050-42ef-beef-2a72969adc5a-kube-api-access-95jv6\" (UniqueName: \"kubernetes.io/projected/846d6de0-2050-42ef-beef-2a72969adc5a-kube-api-access-95jv6\") pod \"846d6de0-2050-42ef-beef-2a72969adc5a\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " Nov 01 22:51:28.485588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:28.485464 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-xks-ca\") pod \"846d6de0-2050-42ef-beef-2a72969adc5a\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " Nov 01 22:51:28.485588 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:28.485492 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-openshift-ca\") pod \"846d6de0-2050-42ef-beef-2a72969adc5a\" (UID: \"846d6de0-2050-42ef-beef-2a72969adc5a\") " Nov 01 22:51:28.485891 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:28.485863 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-openshift-ca" pod "846d6de0-2050-42ef-beef-2a72969adc5a" (UID: "846d6de0-2050-42ef-beef-2a72969adc5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:51:28.487446 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:28.487422 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846d6de0-2050-42ef-beef-2a72969adc5a-kube-api-access-95jv6" pod "846d6de0-2050-42ef-beef-2a72969adc5a" (UID: "846d6de0-2050-42ef-beef-2a72969adc5a"). InnerVolumeSpecName "kube-api-access-95jv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:28.487524 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:28.487474 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-xks-ca" pod "846d6de0-2050-42ef-beef-2a72969adc5a" (UID: "846d6de0-2050-42ef-beef-2a72969adc5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:51:28.586335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:28.586294 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-xks-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:28.586335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:28.586327 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846d6de0-2050-42ef-beef-2a72969adc5a-maas-api-openshift-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:28.586335 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:28.586342 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95jv6\" (UniqueName: \"kubernetes.io/projected/846d6de0-2050-42ef-beef-2a72969adc5a-kube-api-access-95jv6\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:29.214235 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:29.214198 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" event={"ID":"846d6de0-2050-42ef-beef-2a72969adc5a","Type":"ContainerDied","Data":"16eda4a7a0a5790368d46a8826b23c1178d2488bd05426f6f6960f387cfb3ef5"} Nov 01 22:51:29.214413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:29.214252 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16eda4a7a0a5790368d46a8826b23c1178d2488bd05426f6f6960f387cfb3ef5" Nov 01 22:51:29.214413 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:29.214250 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf" Nov 01 22:51:37.073266 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.073228 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b57d74f86-drs46"] Nov 01 22:51:37.075821 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.073434 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7b57d74f86-drs46" podUID="cf4b1b0b-6012-42e7-b4d1-2a7293549ea0" containerName="manager" containerID="cri-o://dee6c5312a0f16ff373d880fe537bfc97f00a167899b1601a158aff9f13eb193" gracePeriod=30 Nov 01 22:51:37.260427 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.260395 6466 generic.go:356] "Generic (PLEG): container finished" podID="cf4b1b0b-6012-42e7-b4d1-2a7293549ea0" containerID="dee6c5312a0f16ff373d880fe537bfc97f00a167899b1601a158aff9f13eb193" exitCode=0 Nov 01 22:51:37.260596 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.260475 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b57d74f86-drs46" event={"ID":"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0","Type":"ContainerDied","Data":"dee6c5312a0f16ff373d880fe537bfc97f00a167899b1601a158aff9f13eb193"} Nov 01 22:51:37.331758 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.331700 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:37.475780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.475747 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-metrics-certs\") pod \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " Nov 01 22:51:37.475780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.475783 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-webhook-cert\") pod \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " Nov 01 22:51:37.476058 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.475831 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-kube-api-access-5sk7t\" (UniqueName: \"kubernetes.io/projected/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-kube-api-access-5sk7t\") pod \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\" (UID: \"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0\") " Nov 01 22:51:37.478219 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.478191 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-metrics-certs" pod "cf4b1b0b-6012-42e7-b4d1-2a7293549ea0" (UID: "cf4b1b0b-6012-42e7-b4d1-2a7293549ea0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:51:37.478531 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.478504 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-webhook-cert" pod "cf4b1b0b-6012-42e7-b4d1-2a7293549ea0" (UID: "cf4b1b0b-6012-42e7-b4d1-2a7293549ea0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:51:37.478606 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.478507 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-kube-api-access-5sk7t" pod "cf4b1b0b-6012-42e7-b4d1-2a7293549ea0" (UID: "cf4b1b0b-6012-42e7-b4d1-2a7293549ea0"). InnerVolumeSpecName "kube-api-access-5sk7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:37.577562 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.577522 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:37.577562 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.577553 6466 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-webhook-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:37.577562 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:37.577564 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sk7t\" (UniqueName: \"kubernetes.io/projected/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0-kube-api-access-5sk7t\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:38.267060 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:38.267024 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b57d74f86-drs46" event={"ID":"cf4b1b0b-6012-42e7-b4d1-2a7293549ea0","Type":"ContainerDied","Data":"68c9582b015706a47cfbdfd6927ab9f6955195212b1d0dcae06a12100f5f1d8a"} Nov 01 22:51:38.267060 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:38.267042 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b57d74f86-drs46" Nov 01 22:51:38.267622 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:38.267136 6466 scope.go:122] "RemoveContainer" containerID="dee6c5312a0f16ff373d880fe537bfc97f00a167899b1601a158aff9f13eb193" Nov 01 22:51:38.289890 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:38.289866 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b57d74f86-drs46"] Nov 01 22:51:38.294642 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:38.294622 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b57d74f86-drs46"] Nov 01 22:51:38.339306 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:38.339279 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4b1b0b-6012-42e7-b4d1-2a7293549ea0" path="/var/lib/kubelet/pods/cf4b1b0b-6012-42e7-b4d1-2a7293549ea0/volumes" Nov 01 22:51:39.547735 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.547700 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b57d74f86-55cvs"] Nov 01 22:51:39.548168 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.548153 6466 cpu_manager.go:414] "containerMap: removing container" podUID="846d6de0-2050-42ef-beef-2a72969adc5a" containerName="revoke-keys" Nov 01 22:51:39.548227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.548168 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846d6de0-2050-42ef-beef-2a72969adc5a" containerName="revoke-keys" Nov 01 22:51:39.548227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.548186 6466 cpu_manager.go:414] "containerMap: removing container" podUID="cf4b1b0b-6012-42e7-b4d1-2a7293549ea0" containerName="manager" Nov 01 22:51:39.548227 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.548192 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4b1b0b-6012-42e7-b4d1-2a7293549ea0" containerName="manager" Nov 01 22:51:39.548321 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.548251 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="846d6de0-2050-42ef-beef-2a72969adc5a" containerName="revoke-keys" Nov 01 22:51:39.548321 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.548262 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4b1b0b-6012-42e7-b4d1-2a7293549ea0" containerName="manager" Nov 01 22:51:39.552825 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.552803 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:39.561000 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.560959 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 22:51:39.561000 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.560983 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 22:51:39.561218 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.561033 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 22:51:39.561218 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.561139 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-m2mcq\"" Nov 01 22:51:39.562152 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.562103 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b57d74f86-55cvs"] Nov 01 22:51:39.697615 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.697581 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30bab32-ebb2-4bc9-93b4-9cd67b593e4f-metrics-certs\") pod \"maas-controller-7b57d74f86-55cvs\" (UID: \"c30bab32-ebb2-4bc9-93b4-9cd67b593e4f\") " pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:39.697786 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.697663 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ljf\" (UniqueName: \"kubernetes.io/projected/c30bab32-ebb2-4bc9-93b4-9cd67b593e4f-kube-api-access-88ljf\") pod \"maas-controller-7b57d74f86-55cvs\" (UID: \"c30bab32-ebb2-4bc9-93b4-9cd67b593e4f\") " pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:39.697786 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.697726 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c30bab32-ebb2-4bc9-93b4-9cd67b593e4f-webhook-cert\") pod \"maas-controller-7b57d74f86-55cvs\" (UID: \"c30bab32-ebb2-4bc9-93b4-9cd67b593e4f\") " pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:39.798420 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.798331 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30bab32-ebb2-4bc9-93b4-9cd67b593e4f-metrics-certs\") pod \"maas-controller-7b57d74f86-55cvs\" (UID: \"c30bab32-ebb2-4bc9-93b4-9cd67b593e4f\") " pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:39.798420 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.798400 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ljf\" (UniqueName: \"kubernetes.io/projected/c30bab32-ebb2-4bc9-93b4-9cd67b593e4f-kube-api-access-88ljf\") pod \"maas-controller-7b57d74f86-55cvs\" (UID: \"c30bab32-ebb2-4bc9-93b4-9cd67b593e4f\") " pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:39.798648 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.798432 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c30bab32-ebb2-4bc9-93b4-9cd67b593e4f-webhook-cert\") pod \"maas-controller-7b57d74f86-55cvs\" (UID: \"c30bab32-ebb2-4bc9-93b4-9cd67b593e4f\") " pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:39.800852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.800826 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c30bab32-ebb2-4bc9-93b4-9cd67b593e4f-metrics-certs\") pod \"maas-controller-7b57d74f86-55cvs\" (UID: \"c30bab32-ebb2-4bc9-93b4-9cd67b593e4f\") " pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:39.800852 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.800844 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c30bab32-ebb2-4bc9-93b4-9cd67b593e4f-webhook-cert\") pod \"maas-controller-7b57d74f86-55cvs\" (UID: \"c30bab32-ebb2-4bc9-93b4-9cd67b593e4f\") " pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:39.806781 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.806761 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ljf\" (UniqueName: \"kubernetes.io/projected/c30bab32-ebb2-4bc9-93b4-9cd67b593e4f-kube-api-access-88ljf\") pod \"maas-controller-7b57d74f86-55cvs\" (UID: \"c30bab32-ebb2-4bc9-93b4-9cd67b593e4f\") " pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:39.867813 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:39.867780 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:40.152128 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:40.152088 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b57d74f86-55cvs"] Nov 01 22:51:40.152842 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:51:40.152813 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30bab32_ebb2_4bc9_93b4_9cd67b593e4f.slice/crio-52f11cab519dc5ef232496077f82c3fb75050692f0655117a20836e5c6bde1f1 WatchSource:0}: Error finding container 52f11cab519dc5ef232496077f82c3fb75050692f0655117a20836e5c6bde1f1: Status 404 returned error can't find the container with id 52f11cab519dc5ef232496077f82c3fb75050692f0655117a20836e5c6bde1f1 Nov 01 22:51:40.279651 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:40.279619 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b57d74f86-55cvs" event={"ID":"c30bab32-ebb2-4bc9-93b4-9cd67b593e4f","Type":"ContainerStarted","Data":"52f11cab519dc5ef232496077f82c3fb75050692f0655117a20836e5c6bde1f1"} Nov 01 22:51:41.287725 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:41.287689 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b57d74f86-55cvs" event={"ID":"c30bab32-ebb2-4bc9-93b4-9cd67b593e4f","Type":"ContainerStarted","Data":"c639b1879db4a185eb5f510a513a584d8df34b982bc7d2f4ed8405ee7027ca5a"} Nov 01 22:51:41.288174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:41.287820 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:51:41.307464 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:41.307412 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b57d74f86-55cvs" podStartSLOduration=1.86633813 podStartE2EDuration="2.307398011s" totalImagesPullingTime="441.059881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:41.3064365 +0000 UTC m=+1705.090875418" watchObservedRunningTime="2026-11-01 22:51:41.307398011 +0000 UTC m=+1705.091836932" Nov 01 22:51:49.280053 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:51:49.280011 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:51:52.295220 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:51:52.295191 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b57d74f86-55cvs" Nov 01 22:52:16.606619 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.606588 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6"] Nov 01 22:52:16.610510 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.610489 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:16.616212 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.616187 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:52:16.616379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.616358 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:52:16.616557 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.616473 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:52:16.616658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.616629 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:52:16.618756 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.618659 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6"] Nov 01 22:52:16.733472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.733429 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcvk\" (UniqueName: \"kubernetes.io/projected/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-kube-api-access-mzcvk\") pod \"maas-api-revoke-sub-c1e271a9-g7sh6\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:16.733658 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.733583 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-g7sh6\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:16.733736 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.733717 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-g7sh6\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:16.834857 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.834829 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-g7sh6\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:16.835042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.834898 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-g7sh6\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:16.835042 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.834930 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcvk\" (UniqueName: \"kubernetes.io/projected/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-kube-api-access-mzcvk\") pod \"maas-api-revoke-sub-c1e271a9-g7sh6\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:16.835583 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.835557 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-g7sh6\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:16.837559 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.837531 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-g7sh6\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:16.843449 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.843427 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcvk\" (UniqueName: \"kubernetes.io/projected/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-kube-api-access-mzcvk\") pod \"maas-api-revoke-sub-c1e271a9-g7sh6\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:16.925234 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:16.925196 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:17.199222 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:17.199046 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6"] Nov 01 22:52:17.201628 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:52:17.201602 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5144e4de_07d3_4ff8_8a03_64cfc4b3b3b8.slice/crio-39671583cf3beb7c89a06486362d1e5a1b304f824016220ab2d87d67eb9b7c02 WatchSource:0}: Error finding container 39671583cf3beb7c89a06486362d1e5a1b304f824016220ab2d87d67eb9b7c02: Status 404 returned error can't find the container with id 39671583cf3beb7c89a06486362d1e5a1b304f824016220ab2d87d67eb9b7c02 Nov 01 22:52:17.530177 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:17.530141 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" event={"ID":"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8","Type":"ContainerStarted","Data":"13f9d6e144d8167c57ddf9876f63ec29994623af0d5ddacc951d53ef5edc96b1"} Nov 01 22:52:17.530348 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:17.530199 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" event={"ID":"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8","Type":"ContainerStarted","Data":"39671583cf3beb7c89a06486362d1e5a1b304f824016220ab2d87d67eb9b7c02"} Nov 01 22:52:17.549097 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:17.549038 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" podStartSLOduration=1.549022247 podStartE2EDuration="1.549022247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:17.547997334 +0000 UTC m=+1741.332436256" watchObservedRunningTime="2026-11-01 22:52:17.549022247 +0000 UTC m=+1741.333461167" Nov 01 22:52:18.536880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:18.536840 6466 generic.go:356] "Generic (PLEG): container finished" podID="5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8" containerID="13f9d6e144d8167c57ddf9876f63ec29994623af0d5ddacc951d53ef5edc96b1" exitCode=0 Nov 01 22:52:18.537363 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:18.536890 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" event={"ID":"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8","Type":"ContainerDied","Data":"13f9d6e144d8167c57ddf9876f63ec29994623af0d5ddacc951d53ef5edc96b1"} Nov 01 22:52:19.280065 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:52:19.280030 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:52:19.680830 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:19.680805 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:19.867481 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:19.867390 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-xks-ca\") pod \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " Nov 01 22:52:19.867674 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:19.867504 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-openshift-ca\") pod \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " Nov 01 22:52:19.867674 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:19.867592 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-kube-api-access-mzcvk\" (UniqueName: \"kubernetes.io/projected/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-kube-api-access-mzcvk\") pod \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\" (UID: \"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8\") " Nov 01 22:52:19.867880 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:19.867838 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-openshift-ca" pod "5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8" (UID: "5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:52:19.869502 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:19.869477 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-xks-ca" pod "5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8" (UID: "5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:52:19.869673 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:19.869650 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-kube-api-access-mzcvk" pod "5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8" (UID: "5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8"). InnerVolumeSpecName "kube-api-access-mzcvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:19.968561 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:19.968525 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcvk\" (UniqueName: \"kubernetes.io/projected/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-kube-api-access-mzcvk\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:19.968561 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:19.968555 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-xks-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:19.968561 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:19.968565 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8-maas-api-openshift-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:20.549866 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:20.549835 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" Nov 01 22:52:20.550066 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:20.549884 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6" event={"ID":"5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8","Type":"ContainerDied","Data":"39671583cf3beb7c89a06486362d1e5a1b304f824016220ab2d87d67eb9b7c02"} Nov 01 22:52:20.550066 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:20.549933 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39671583cf3beb7c89a06486362d1e5a1b304f824016220ab2d87d67eb9b7c02" Nov 01 22:52:35.260448 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.260414 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs"] Nov 01 22:52:35.261026 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.260956 6466 cpu_manager.go:414] "containerMap: removing container" podUID="5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8" containerName="revoke-keys" Nov 01 22:52:35.261026 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.260970 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8" containerName="revoke-keys" Nov 01 22:52:35.261165 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.261052 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8" containerName="revoke-keys" Nov 01 22:52:35.265537 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.265522 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:35.272876 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.272856 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:52:35.273017 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.272996 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:52:35.273101 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.273048 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:52:35.273199 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.273151 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:52:35.275341 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.275316 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs"] Nov 01 22:52:35.286765 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.286739 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hnmqs\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:35.286917 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.286816 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dw6j\" (UniqueName: \"kubernetes.io/projected/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-kube-api-access-9dw6j\") pod \"maas-api-revoke-sub-2e1f5b5f-hnmqs\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:35.286978 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.286948 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hnmqs\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:35.388266 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.388229 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hnmqs\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:35.388465 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.388292 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hnmqs\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:35.388465 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.388337 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dw6j\" (UniqueName: \"kubernetes.io/projected/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-kube-api-access-9dw6j\") pod \"maas-api-revoke-sub-2e1f5b5f-hnmqs\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:35.389062 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.389037 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hnmqs\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:35.390711 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.390685 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hnmqs\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:35.397035 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.397008 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dw6j\" (UniqueName: \"kubernetes.io/projected/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-kube-api-access-9dw6j\") pod \"maas-api-revoke-sub-2e1f5b5f-hnmqs\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:35.582976 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.582896 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:35.858259 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:35.858233 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs"] Nov 01 22:52:35.860312 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:52:35.860284 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f56238_8ae3_45bf_9882_c6bee7bfe06c.slice/crio-31f368566ea961895f76a3094319570ab10295b864100bac977f901f06e273a1 WatchSource:0}: Error finding container 31f368566ea961895f76a3094319570ab10295b864100bac977f901f06e273a1: Status 404 returned error can't find the container with id 31f368566ea961895f76a3094319570ab10295b864100bac977f901f06e273a1 Nov 01 22:52:36.664167 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:36.664128 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" event={"ID":"b5f56238-8ae3-45bf-9882-c6bee7bfe06c","Type":"ContainerStarted","Data":"155f46c61f846ce7b9d479c014b3be5027988e5fd83818507fba5fda4d2dc278"} Nov 01 22:52:36.664570 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:36.664192 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" event={"ID":"b5f56238-8ae3-45bf-9882-c6bee7bfe06c","Type":"ContainerStarted","Data":"31f368566ea961895f76a3094319570ab10295b864100bac977f901f06e273a1"} Nov 01 22:52:36.682150 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:36.682076 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" podStartSLOduration=1.682062992 podStartE2EDuration="1.682062992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:36.680705552 +0000 UTC m=+1760.465144474" watchObservedRunningTime="2026-11-01 22:52:36.682062992 +0000 UTC m=+1760.466501912" Nov 01 22:52:37.670217 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:37.670182 6466 generic.go:356] "Generic (PLEG): container finished" podID="b5f56238-8ae3-45bf-9882-c6bee7bfe06c" containerID="155f46c61f846ce7b9d479c014b3be5027988e5fd83818507fba5fda4d2dc278" exitCode=0 Nov 01 22:52:37.670650 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:37.670254 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" event={"ID":"b5f56238-8ae3-45bf-9882-c6bee7bfe06c","Type":"ContainerDied","Data":"155f46c61f846ce7b9d479c014b3be5027988e5fd83818507fba5fda4d2dc278"} Nov 01 22:52:38.819346 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:38.819316 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:38.926604 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:38.926514 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-xks-ca\") pod \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " Nov 01 22:52:38.926765 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:38.926672 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-openshift-ca\") pod \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " Nov 01 22:52:38.926765 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:38.926718 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-kube-api-access-9dw6j\" (UniqueName: \"kubernetes.io/projected/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-kube-api-access-9dw6j\") pod \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\" (UID: \"b5f56238-8ae3-45bf-9882-c6bee7bfe06c\") " Nov 01 22:52:38.927051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:38.927024 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-openshift-ca" pod "b5f56238-8ae3-45bf-9882-c6bee7bfe06c" (UID: "b5f56238-8ae3-45bf-9882-c6bee7bfe06c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:52:38.928775 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:38.928752 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-kube-api-access-9dw6j" pod "b5f56238-8ae3-45bf-9882-c6bee7bfe06c" (UID: "b5f56238-8ae3-45bf-9882-c6bee7bfe06c"). InnerVolumeSpecName "kube-api-access-9dw6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:38.928846 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:38.928755 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-xks-ca" pod "b5f56238-8ae3-45bf-9882-c6bee7bfe06c" (UID: "b5f56238-8ae3-45bf-9882-c6bee7bfe06c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:52:39.028051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:39.028019 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-openshift-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:39.028051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:39.028047 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dw6j\" (UniqueName: \"kubernetes.io/projected/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-kube-api-access-9dw6j\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:39.028285 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:39.028059 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f56238-8ae3-45bf-9882-c6bee7bfe06c-maas-api-xks-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:39.681162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:39.681103 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" Nov 01 22:52:39.681162 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:39.681146 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs" event={"ID":"b5f56238-8ae3-45bf-9882-c6bee7bfe06c","Type":"ContainerDied","Data":"31f368566ea961895f76a3094319570ab10295b864100bac977f901f06e273a1"} Nov 01 22:52:39.681379 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:39.681194 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31f368566ea961895f76a3094319570ab10295b864100bac977f901f06e273a1" Nov 01 22:52:46.232268 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.232232 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd"] Nov 01 22:52:46.232855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.232724 6466 cpu_manager.go:414] "containerMap: removing container" podUID="b5f56238-8ae3-45bf-9882-c6bee7bfe06c" containerName="revoke-keys" Nov 01 22:52:46.232855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.232739 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f56238-8ae3-45bf-9882-c6bee7bfe06c" containerName="revoke-keys" Nov 01 22:52:46.232855 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.232819 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f56238-8ae3-45bf-9882-c6bee7bfe06c" containerName="revoke-keys" Nov 01 22:52:46.236214 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.236188 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:46.242369 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.242340 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:52:46.242898 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.242874 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:52:46.243050 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.242957 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:52:46.244153 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.244136 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:52:46.246734 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.245652 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd"] Nov 01 22:52:46.302395 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.302352 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jb4t\" (UniqueName: \"kubernetes.io/projected/219f7ade-afc6-4502-9347-2527b8245a19-kube-api-access-4jb4t\") pod \"maas-api-revoke-sub-ec843018-rhxbd\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:46.302582 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.302413 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rhxbd\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:46.302582 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.302558 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rhxbd\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:46.403303 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.403269 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rhxbd\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:46.403479 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.403382 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jb4t\" (UniqueName: \"kubernetes.io/projected/219f7ade-afc6-4502-9347-2527b8245a19-kube-api-access-4jb4t\") pod \"maas-api-revoke-sub-ec843018-rhxbd\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:46.403479 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.403412 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rhxbd\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:46.403932 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.403910 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rhxbd\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:46.405772 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.405744 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rhxbd\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:46.411409 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.411385 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jb4t\" (UniqueName: \"kubernetes.io/projected/219f7ade-afc6-4502-9347-2527b8245a19-kube-api-access-4jb4t\") pod \"maas-api-revoke-sub-ec843018-rhxbd\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:46.554036 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.553951 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:46.842069 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:46.842005 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd"] Nov 01 22:52:46.844472 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:52:46.844433 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219f7ade_afc6_4502_9347_2527b8245a19.slice/crio-e6dd571f5619ae374b85195ab0f17b653f0bad6cbcfb29c94d6bf1d1dbd9ffaf WatchSource:0}: Error finding container e6dd571f5619ae374b85195ab0f17b653f0bad6cbcfb29c94d6bf1d1dbd9ffaf: Status 404 returned error can't find the container with id e6dd571f5619ae374b85195ab0f17b653f0bad6cbcfb29c94d6bf1d1dbd9ffaf Nov 01 22:52:47.722570 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:47.722535 6466 generic.go:356] "Generic (PLEG): container finished" podID="219f7ade-afc6-4502-9347-2527b8245a19" containerID="01f9aa5b7640921469e7ef5a7b94fe7bc71fe823682fdfe9574299205073f9b1" exitCode=0 Nov 01 22:52:47.722949 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:47.722615 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" event={"ID":"219f7ade-afc6-4502-9347-2527b8245a19","Type":"ContainerDied","Data":"01f9aa5b7640921469e7ef5a7b94fe7bc71fe823682fdfe9574299205073f9b1"} Nov 01 22:52:47.722949 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:47.722690 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" event={"ID":"219f7ade-afc6-4502-9347-2527b8245a19","Type":"ContainerStarted","Data":"e6dd571f5619ae374b85195ab0f17b653f0bad6cbcfb29c94d6bf1d1dbd9ffaf"} Nov 01 22:52:48.930953 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:48.930919 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:52:49.033154 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.033032 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-xks-ca\") pod \"219f7ade-afc6-4502-9347-2527b8245a19\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " Nov 01 22:52:49.033349 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.033267 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-openshift-ca\") pod \"219f7ade-afc6-4502-9347-2527b8245a19\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " Nov 01 22:52:49.033425 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.033347 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219f7ade-afc6-4502-9347-2527b8245a19-kube-api-access-4jb4t\" (UniqueName: \"kubernetes.io/projected/219f7ade-afc6-4502-9347-2527b8245a19-kube-api-access-4jb4t\") pod \"219f7ade-afc6-4502-9347-2527b8245a19\" (UID: \"219f7ade-afc6-4502-9347-2527b8245a19\") " Nov 01 22:52:49.033816 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.033781 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-openshift-ca" pod "219f7ade-afc6-4502-9347-2527b8245a19" (UID: "219f7ade-afc6-4502-9347-2527b8245a19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:52:49.035836 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.035812 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-xks-ca" pod "219f7ade-afc6-4502-9347-2527b8245a19" (UID: "219f7ade-afc6-4502-9347-2527b8245a19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:52:49.036352 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.036327 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219f7ade-afc6-4502-9347-2527b8245a19-kube-api-access-4jb4t" pod "219f7ade-afc6-4502-9347-2527b8245a19" (UID: "219f7ade-afc6-4502-9347-2527b8245a19"). InnerVolumeSpecName "kube-api-access-4jb4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:49.135891 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.135799 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-openshift-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:49.135891 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.135846 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jb4t\" (UniqueName: \"kubernetes.io/projected/219f7ade-afc6-4502-9347-2527b8245a19-kube-api-access-4jb4t\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:49.135891 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.135863 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219f7ade-afc6-4502-9347-2527b8245a19-maas-api-xks-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:49.419846 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:52:49.419702 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:52:49.752027 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.751930 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" event={"ID":"219f7ade-afc6-4502-9347-2527b8245a19","Type":"ContainerDied","Data":"e6dd571f5619ae374b85195ab0f17b653f0bad6cbcfb29c94d6bf1d1dbd9ffaf"} Nov 01 22:52:49.752326 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.752304 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6dd571f5619ae374b85195ab0f17b653f0bad6cbcfb29c94d6bf1d1dbd9ffaf" Nov 01 22:52:49.752522 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:52:49.752508 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd" Nov 01 22:53:12.319134 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.319070 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r"] Nov 01 22:53:12.319780 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.319758 6466 cpu_manager.go:414] "containerMap: removing container" podUID="219f7ade-afc6-4502-9347-2527b8245a19" containerName="revoke-keys" Nov 01 22:53:12.319871 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.319783 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219f7ade-afc6-4502-9347-2527b8245a19" containerName="revoke-keys" Nov 01 22:53:12.319959 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.319945 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="219f7ade-afc6-4502-9347-2527b8245a19" containerName="revoke-keys" Nov 01 22:53:12.323245 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.323224 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:12.330549 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.329763 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:53:12.330549 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.329808 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:53:12.330549 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.329888 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:53:12.332154 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.331863 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:53:12.343660 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.343633 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r"] Nov 01 22:53:12.473692 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.473652 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e404dd-456f-414d-8018-3e8585163874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9pm4r\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:12.473878 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.473713 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e404dd-456f-414d-8018-3e8585163874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9pm4r\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:12.473927 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.473878 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p87bw\" (UniqueName: \"kubernetes.io/projected/50e404dd-456f-414d-8018-3e8585163874-kube-api-access-p87bw\") pod \"maas-api-revoke-sub-046e1343-9pm4r\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:12.575399 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.575303 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e404dd-456f-414d-8018-3e8585163874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9pm4r\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:12.575399 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.575383 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e404dd-456f-414d-8018-3e8585163874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9pm4r\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:12.575641 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.575512 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p87bw\" (UniqueName: \"kubernetes.io/projected/50e404dd-456f-414d-8018-3e8585163874-kube-api-access-p87bw\") pod \"maas-api-revoke-sub-046e1343-9pm4r\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:12.576222 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.576196 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e404dd-456f-414d-8018-3e8585163874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9pm4r\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:12.577956 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.577930 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e404dd-456f-414d-8018-3e8585163874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9pm4r\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:12.584144 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.584123 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p87bw\" (UniqueName: \"kubernetes.io/projected/50e404dd-456f-414d-8018-3e8585163874-kube-api-access-p87bw\") pod \"maas-api-revoke-sub-046e1343-9pm4r\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:12.640183 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.640151 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:12.915719 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.915640 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r"] Nov 01 22:53:12.918248 ip-10-0-134-88 kubenswrapper[6466]: W1101 22:53:12.918215 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e404dd_456f_414d_8018_3e8585163874.slice/crio-ecf5db12ae93debd07131d913c58cc641e85f29a398b86cad6877662323f844d WatchSource:0}: Error finding container ecf5db12ae93debd07131d913c58cc641e85f29a398b86cad6877662323f844d: Status 404 returned error can't find the container with id ecf5db12ae93debd07131d913c58cc641e85f29a398b86cad6877662323f844d Nov 01 22:53:12.984131 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:12.984078 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" event={"ID":"50e404dd-456f-414d-8018-3e8585163874","Type":"ContainerStarted","Data":"ecf5db12ae93debd07131d913c58cc641e85f29a398b86cad6877662323f844d"} Nov 01 22:53:13.990527 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:13.990494 6466 generic.go:356] "Generic (PLEG): container finished" podID="50e404dd-456f-414d-8018-3e8585163874" containerID="2825a2b8e164e6e90ea26a5e4298b1c99f07e9fa6839e3a5cb6a6cf2486681f5" exitCode=0 Nov 01 22:53:13.990933 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:13.990584 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" event={"ID":"50e404dd-456f-414d-8018-3e8585163874","Type":"ContainerDied","Data":"2825a2b8e164e6e90ea26a5e4298b1c99f07e9fa6839e3a5cb6a6cf2486681f5"} Nov 01 22:53:15.146226 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:15.146204 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:15.306910 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:15.306826 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e404dd-456f-414d-8018-3e8585163874-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e404dd-456f-414d-8018-3e8585163874-maas-api-xks-ca\") pod \"50e404dd-456f-414d-8018-3e8585163874\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " Nov 01 22:53:15.306910 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:15.306871 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e404dd-456f-414d-8018-3e8585163874-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e404dd-456f-414d-8018-3e8585163874-maas-api-openshift-ca\") pod \"50e404dd-456f-414d-8018-3e8585163874\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " Nov 01 22:53:15.307174 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:15.307020 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e404dd-456f-414d-8018-3e8585163874-kube-api-access-p87bw\" (UniqueName: \"kubernetes.io/projected/50e404dd-456f-414d-8018-3e8585163874-kube-api-access-p87bw\") pod \"50e404dd-456f-414d-8018-3e8585163874\" (UID: \"50e404dd-456f-414d-8018-3e8585163874\") " Nov 01 22:53:15.307292 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:15.307268 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e404dd-456f-414d-8018-3e8585163874-maas-api-openshift-ca" pod "50e404dd-456f-414d-8018-3e8585163874" (UID: "50e404dd-456f-414d-8018-3e8585163874"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:53:15.309051 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:15.309027 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e404dd-456f-414d-8018-3e8585163874-kube-api-access-p87bw" pod "50e404dd-456f-414d-8018-3e8585163874" (UID: "50e404dd-456f-414d-8018-3e8585163874"). InnerVolumeSpecName "kube-api-access-p87bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:53:15.309159 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:15.309030 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e404dd-456f-414d-8018-3e8585163874-maas-api-xks-ca" pod "50e404dd-456f-414d-8018-3e8585163874" (UID: "50e404dd-456f-414d-8018-3e8585163874"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:53:15.407791 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:15.407759 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p87bw\" (UniqueName: \"kubernetes.io/projected/50e404dd-456f-414d-8018-3e8585163874-kube-api-access-p87bw\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:15.407791 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:15.407787 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e404dd-456f-414d-8018-3e8585163874-maas-api-xks-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:15.408015 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:15.407801 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e404dd-456f-414d-8018-3e8585163874-maas-api-openshift-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:16.001472 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:16.001442 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" Nov 01 22:53:16.001657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:16.001478 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r" event={"ID":"50e404dd-456f-414d-8018-3e8585163874","Type":"ContainerDied","Data":"ecf5db12ae93debd07131d913c58cc641e85f29a398b86cad6877662323f844d"} Nov 01 22:53:16.001657 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:53:16.001528 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecf5db12ae93debd07131d913c58cc641e85f29a398b86cad6877662323f844d" Nov 01 22:53:19.280833 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:53:19.280791 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:53:49.281027 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:53:49.280991 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:54:19.280121 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:54:19.280071 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:54:49.281803 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:54:49.281760 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:55:19.280464 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:55:19.280382 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:55:40.037447 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:55:40.037410 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn"] Nov 01 22:55:40.042843 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:55:40.042815 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a0fa409-65kmn"] Nov 01 22:55:40.340456 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:55:40.340379 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d9aead-f494-428f-954a-9f4544bdb67d" path="/var/lib/kubelet/pods/41d9aead-f494-428f-954a-9f4544bdb67d/volumes" Nov 01 22:55:49.281211 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:55:49.281167 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:56:19.280499 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:56:19.280460 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:56:29.027665 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:56:29.027583 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf"] Nov 01 22:56:29.031195 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:56:29.031166 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mkcjf"] Nov 01 22:56:30.339550 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:56:30.339515 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846d6de0-2050-42ef-beef-2a72969adc5a" path="/var/lib/kubelet/pods/846d6de0-2050-42ef-beef-2a72969adc5a/volumes" Nov 01 22:56:49.280850 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:56:49.280808 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:57:19.279950 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:57:19.279910 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:57:20.020654 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:57:20.020617 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6"] Nov 01 22:57:20.022984 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:57:20.022954 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-g7sh6"] Nov 01 22:57:20.339339 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:57:20.339250 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8" path="/var/lib/kubelet/pods/5144e4de-07d3-4ff8-8a03-64cfc4b3b3b8/volumes" Nov 01 22:57:22.006437 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:57:22.006406 6466 scope.go:122] "RemoveContainer" containerID="88895663595d91dd79c812302d19e8c4f1480fae2bbbf9b35218ed212d2e6013" Nov 01 22:57:22.015467 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:57:22.015446 6466 scope.go:122] "RemoveContainer" containerID="2ad302274dc5b1d73fc602d1dbbf7b9ad8215c8a8b8a5d224abb8e81aa26236a" Nov 01 22:57:39.028362 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:57:39.028320 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs"] Nov 01 22:57:39.029782 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:57:39.029759 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hnmqs"] Nov 01 22:57:40.339485 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:57:40.339452 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f56238-8ae3-45bf-9882-c6bee7bfe06c" path="/var/lib/kubelet/pods/b5f56238-8ae3-45bf-9882-c6bee7bfe06c/volumes" Nov 01 22:57:49.021673 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:57:49.021629 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd"] Nov 01 22:57:49.023865 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:57:49.023845 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rhxbd"] Nov 01 22:57:49.280076 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:57:49.279980 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:57:50.339677 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:57:50.339634 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219f7ade-afc6-4502-9347-2527b8245a19" path="/var/lib/kubelet/pods/219f7ade-afc6-4502-9347-2527b8245a19/volumes" Nov 01 22:58:16.023491 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:58:16.023416 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r"] Nov 01 22:58:16.027370 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:58:16.027341 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9pm4r"] Nov 01 22:58:16.339983 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:58:16.339891 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e404dd-456f-414d-8018-3e8585163874" path="/var/lib/kubelet/pods/50e404dd-456f-414d-8018-3e8585163874/volumes" Nov 01 22:58:19.279787 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:58:19.279753 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:58:22.042552 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:58:22.042515 6466 scope.go:122] "RemoveContainer" containerID="13f9d6e144d8167c57ddf9876f63ec29994623af0d5ddacc951d53ef5edc96b1" Nov 01 22:58:22.051297 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:58:22.051271 6466 scope.go:122] "RemoveContainer" containerID="b6e9d9e014d72e72e0eba0e32bd4967ddc6eb41b42505cd8dab1555a6c3d6f26" Nov 01 22:58:49.280556 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:58:49.280516 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:59:19.281177 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:59:19.281139 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 22:59:22.075400 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:59:22.075367 6466 scope.go:122] "RemoveContainer" containerID="155f46c61f846ce7b9d479c014b3be5027988e5fd83818507fba5fda4d2dc278" Nov 01 22:59:22.083851 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:59:22.083826 6466 scope.go:122] "RemoveContainer" containerID="2825a2b8e164e6e90ea26a5e4298b1c99f07e9fa6839e3a5cb6a6cf2486681f5" Nov 01 22:59:22.091847 ip-10-0-134-88 kubenswrapper[6466]: I1101 22:59:22.091821 6466 scope.go:122] "RemoveContainer" containerID="01f9aa5b7640921469e7ef5a7b94fe7bc71fe823682fdfe9574299205073f9b1" Nov 01 22:59:49.282746 ip-10-0-134-88 kubenswrapper[6466]: E1101 22:59:49.282696 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:00:19.280985 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:00:19.280946 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:00:49.279994 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:00:49.279955 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:01:10.377375 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:10.377330 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59"] Nov 01 23:01:10.378309 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:10.378250 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" containerID="cri-o://74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9" gracePeriod=30 Nov 01 23:01:10.379568 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:10.379473 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2"] Nov 01 23:01:10.379785 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:10.379747 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" containerID="cri-o://9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8" gracePeriod=30 Nov 01 23:01:10.381560 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:10.381533 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc"] Nov 01 23:01:10.381787 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:10.381745 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" containerID="cri-o://3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5" gracePeriod=30 Nov 01 23:01:10.383171 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:10.383148 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6"] Nov 01 23:01:10.383342 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:10.383310 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" containerID="cri-o://886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f" gracePeriod=30 Nov 01 23:01:10.918127 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:10.918074 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9gn59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:01:10.918310 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:10.918153 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:01:15.645708 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.645678 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 23:01:15.695833 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.695807 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 23:01:15.722236 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.721858 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 23:01:15.730534 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.730505 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 23:01:15.741040 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.741015 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-kube-api-access-gnw92\" (UniqueName: \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-kube-api-access-gnw92\") pod \"45719f2c-6797-450c-8089-e90c493e3b3a\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " Nov 01 23:01:15.741206 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.741055 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-token\" (UniqueName: \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-token\") pod \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " Nov 01 23:01:15.741206 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.741092 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-socket\") pod \"45719f2c-6797-450c-8089-e90c493e3b3a\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " Nov 01 23:01:15.741206 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.741145 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-kube-api-access-p6mlr\" (UniqueName: \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-kube-api-access-p6mlr\") pod \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " Nov 01 23:01:15.741661 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.741434 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-istio-token\" (UniqueName: \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-istio-token\") pod \"45719f2c-6797-450c-8089-e90c493e3b3a\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " Nov 01 23:01:15.741661 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.741476 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-socket\") pod \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " Nov 01 23:01:15.742254 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.741842 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-token\" (UniqueName: \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-token\") pod \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " Nov 01 23:01:15.742381 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742358 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-data\") pod \"45719f2c-6797-450c-8089-e90c493e3b3a\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " Nov 01 23:01:15.742484 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742463 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-certs\") pod \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " Nov 01 23:01:15.742550 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742499 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-certs\") pod \"45719f2c-6797-450c-8089-e90c493e3b3a\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " Nov 01 23:01:15.742550 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742527 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-envoy\") pod \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " Nov 01 23:01:15.742628 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742557 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-data\") pod \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " Nov 01 23:01:15.742628 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742584 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-credential-socket\") pod \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " Nov 01 23:01:15.742628 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742612 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-envoy\") pod \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742653 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-data\") pod \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742689 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-credential-socket\") pod \"45719f2c-6797-450c-8089-e90c493e3b3a\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742726 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-kube-api-access-9fs2b\" (UniqueName: \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-kube-api-access-9fs2b\") pod \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742757 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-credential-socket\") pod \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742815 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-podinfo\") pod \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742851 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-podinfo\") pod \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742883 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istiod-ca-cert\") pod \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\" (UID: \"b9169aa9-28c9-4698-8d11-8b0d232d81a0\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742917 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-socket\") pod \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.742954 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45719f2c-6797-450c-8089-e90c493e3b3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45719f2c-6797-450c-8089-e90c493e3b3a-istiod-ca-cert\") pod \"45719f2c-6797-450c-8089-e90c493e3b3a\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.743010 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-certs\") pod \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.743054 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-envoy\") pod \"45719f2c-6797-450c-8089-e90c493e3b3a\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.743084 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istiod-ca-cert\") pod \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\" (UID: \"7aeac0ad-04ea-4c1b-b273-a0d708cc552f\") " Nov 01 23:01:15.743159 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.743157 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45719f2c-6797-450c-8089-e90c493e3b3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45719f2c-6797-450c-8089-e90c493e3b3a-istio-podinfo\") pod \"45719f2c-6797-450c-8089-e90c493e3b3a\" (UID: \"45719f2c-6797-450c-8089-e90c493e3b3a\") " Nov 01 23:01:15.744684 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.744651 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45719f2c-6797-450c-8089-e90c493e3b3a-istiod-ca-cert" pod "45719f2c-6797-450c-8089-e90c493e3b3a" (UID: "45719f2c-6797-450c-8089-e90c493e3b3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:15.745144 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.745049 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-data" pod "7aeac0ad-04ea-4c1b-b273-a0d708cc552f" (UID: "7aeac0ad-04ea-4c1b-b273-a0d708cc552f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.745394 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.745237 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-credential-socket" pod "7aeac0ad-04ea-4c1b-b273-a0d708cc552f" (UID: "7aeac0ad-04ea-4c1b-b273-a0d708cc552f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.745631 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.745501 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-certs" pod "7aeac0ad-04ea-4c1b-b273-a0d708cc552f" (UID: "7aeac0ad-04ea-4c1b-b273-a0d708cc552f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.746195 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.745718 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-data" pod "b9169aa9-28c9-4698-8d11-8b0d232d81a0" (UID: "b9169aa9-28c9-4698-8d11-8b0d232d81a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.746860 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.746570 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-credential-socket" pod "b9169aa9-28c9-4698-8d11-8b0d232d81a0" (UID: "b9169aa9-28c9-4698-8d11-8b0d232d81a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.752743 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.748369 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-credential-socket" pod "45719f2c-6797-450c-8089-e90c493e3b3a" (UID: "45719f2c-6797-450c-8089-e90c493e3b3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.752743 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.748424 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istiod-ca-cert" pod "b9169aa9-28c9-4698-8d11-8b0d232d81a0" (UID: "b9169aa9-28c9-4698-8d11-8b0d232d81a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:15.752743 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.748549 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-socket" pod "b9169aa9-28c9-4698-8d11-8b0d232d81a0" (UID: "b9169aa9-28c9-4698-8d11-8b0d232d81a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.752743 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.748595 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-data" pod "45719f2c-6797-450c-8089-e90c493e3b3a" (UID: "45719f2c-6797-450c-8089-e90c493e3b3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.752743 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.748633 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-socket" pod "45719f2c-6797-450c-8089-e90c493e3b3a" (UID: "45719f2c-6797-450c-8089-e90c493e3b3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.752743 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.748727 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-certs" pod "b9169aa9-28c9-4698-8d11-8b0d232d81a0" (UID: "b9169aa9-28c9-4698-8d11-8b0d232d81a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.752743 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.748769 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istiod-ca-cert" pod "7aeac0ad-04ea-4c1b-b273-a0d708cc552f" (UID: "7aeac0ad-04ea-4c1b-b273-a0d708cc552f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:15.752743 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.749028 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-token" pod "7aeac0ad-04ea-4c1b-b273-a0d708cc552f" (UID: "7aeac0ad-04ea-4c1b-b273-a0d708cc552f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:15.752743 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.749100 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-kube-api-access-gnw92" pod "45719f2c-6797-450c-8089-e90c493e3b3a" (UID: "45719f2c-6797-450c-8089-e90c493e3b3a"). InnerVolumeSpecName "kube-api-access-gnw92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:15.752743 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.749469 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-socket" pod "7aeac0ad-04ea-4c1b-b273-a0d708cc552f" (UID: "7aeac0ad-04ea-4c1b-b273-a0d708cc552f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.752743 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.751954 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-certs" pod "45719f2c-6797-450c-8089-e90c493e3b3a" (UID: "45719f2c-6797-450c-8089-e90c493e3b3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.771032 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.770987 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-istio-token" pod "45719f2c-6797-450c-8089-e90c493e3b3a" (UID: "45719f2c-6797-450c-8089-e90c493e3b3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:15.771520 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.771399 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-kube-api-access-p6mlr" pod "b9169aa9-28c9-4698-8d11-8b0d232d81a0" (UID: "b9169aa9-28c9-4698-8d11-8b0d232d81a0"). InnerVolumeSpecName "kube-api-access-p6mlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:15.771520 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.771401 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-envoy" pod "b9169aa9-28c9-4698-8d11-8b0d232d81a0" (UID: "b9169aa9-28c9-4698-8d11-8b0d232d81a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.771520 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.771422 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-kube-api-access-9fs2b" pod "7aeac0ad-04ea-4c1b-b273-a0d708cc552f" (UID: "7aeac0ad-04ea-4c1b-b273-a0d708cc552f"). InnerVolumeSpecName "kube-api-access-9fs2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:15.771520 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.771396 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45719f2c-6797-450c-8089-e90c493e3b3a-istio-podinfo" pod "45719f2c-6797-450c-8089-e90c493e3b3a" (UID: "45719f2c-6797-450c-8089-e90c493e3b3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:15.771520 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.771417 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-envoy" pod "45719f2c-6797-450c-8089-e90c493e3b3a" (UID: "45719f2c-6797-450c-8089-e90c493e3b3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.771520 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.771454 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-token" pod "b9169aa9-28c9-4698-8d11-8b0d232d81a0" (UID: "b9169aa9-28c9-4698-8d11-8b0d232d81a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:15.771520 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.771470 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-podinfo" pod "7aeac0ad-04ea-4c1b-b273-a0d708cc552f" (UID: "7aeac0ad-04ea-4c1b-b273-a0d708cc552f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:15.771520 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.771496 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-envoy" pod "7aeac0ad-04ea-4c1b-b273-a0d708cc552f" (UID: "7aeac0ad-04ea-4c1b-b273-a0d708cc552f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.773028 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.772996 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-podinfo" pod "b9169aa9-28c9-4698-8d11-8b0d232d81a0" (UID: "b9169aa9-28c9-4698-8d11-8b0d232d81a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:15.845306 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845266 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-credential-socket\") pod \"d9d0e2c6-68dd-486b-a461-1564218a792b\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " Nov 01 23:01:15.845495 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845348 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-envoy\") pod \"d9d0e2c6-68dd-486b-a461-1564218a792b\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " Nov 01 23:01:15.845495 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845374 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-token\" (UniqueName: \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-token\") pod \"d9d0e2c6-68dd-486b-a461-1564218a792b\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " Nov 01 23:01:15.845495 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845393 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-kube-api-access-sqtn6\" (UniqueName: \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-kube-api-access-sqtn6\") pod \"d9d0e2c6-68dd-486b-a461-1564218a792b\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " Nov 01 23:01:15.845495 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845425 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9d0e2c6-68dd-486b-a461-1564218a792b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d0e2c6-68dd-486b-a461-1564218a792b-istiod-ca-cert\") pod \"d9d0e2c6-68dd-486b-a461-1564218a792b\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " Nov 01 23:01:15.845738 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845560 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-podinfo\") pod \"d9d0e2c6-68dd-486b-a461-1564218a792b\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " Nov 01 23:01:15.845738 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845625 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-credential-socket" pod "d9d0e2c6-68dd-486b-a461-1564218a792b" (UID: "d9d0e2c6-68dd-486b-a461-1564218a792b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.845738 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845691 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-socket\") pod \"d9d0e2c6-68dd-486b-a461-1564218a792b\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " Nov 01 23:01:15.845738 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845728 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-data\") pod \"d9d0e2c6-68dd-486b-a461-1564218a792b\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " Nov 01 23:01:15.845962 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845773 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-certs\") pod \"d9d0e2c6-68dd-486b-a461-1564218a792b\" (UID: \"d9d0e2c6-68dd-486b-a461-1564218a792b\") " Nov 01 23:01:15.845962 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845901 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d0e2c6-68dd-486b-a461-1564218a792b-istiod-ca-cert" pod "d9d0e2c6-68dd-486b-a461-1564218a792b" (UID: "d9d0e2c6-68dd-486b-a461-1564218a792b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:15.845962 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.845928 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-socket" pod "d9d0e2c6-68dd-486b-a461-1564218a792b" (UID: "d9d0e2c6-68dd-486b-a461-1564218a792b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.846149 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846037 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-data" pod "d9d0e2c6-68dd-486b-a461-1564218a792b" (UID: "d9d0e2c6-68dd-486b-a461-1564218a792b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.846217 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846178 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-certs" pod "d9d0e2c6-68dd-486b-a461-1564218a792b" (UID: "d9d0e2c6-68dd-486b-a461-1564218a792b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.846357 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846338 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846441 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846363 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846441 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846381 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846441 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846397 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846441 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846411 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846441 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846425 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846441 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846438 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846453 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846467 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846483 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846498 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846513 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846526 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846539 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846553 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846583 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846599 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fs2b\" (UniqueName: \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-kube-api-access-9fs2b\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846615 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9169aa9-28c9-4698-8d11-8b0d232d81a0-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846631 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846645 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846659 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9169aa9-28c9-4698-8d11-8b0d232d81a0-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846674 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846689 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45719f2c-6797-450c-8089-e90c493e3b3a-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846703 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846714 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846727 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9d0e2c6-68dd-486b-a461-1564218a792b-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846742 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846755 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45719f2c-6797-450c-8089-e90c493e3b3a-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846769 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnw92\" (UniqueName: \"kubernetes.io/projected/45719f2c-6797-450c-8089-e90c493e3b3a-kube-api-access-gnw92\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.846779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846783 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aeac0ad-04ea-4c1b-b273-a0d708cc552f-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.848033 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846798 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45719f2c-6797-450c-8089-e90c493e3b3a-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.848033 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.846812 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6mlr\" (UniqueName: \"kubernetes.io/projected/b9169aa9-28c9-4698-8d11-8b0d232d81a0-kube-api-access-p6mlr\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.848033 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.847822 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-podinfo" pod "d9d0e2c6-68dd-486b-a461-1564218a792b" (UID: "d9d0e2c6-68dd-486b-a461-1564218a792b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:15.848033 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.847908 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-envoy" pod "d9d0e2c6-68dd-486b-a461-1564218a792b" (UID: "d9d0e2c6-68dd-486b-a461-1564218a792b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.848033 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.847908 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-kube-api-access-sqtn6" pod "d9d0e2c6-68dd-486b-a461-1564218a792b" (UID: "d9d0e2c6-68dd-486b-a461-1564218a792b"). InnerVolumeSpecName "kube-api-access-sqtn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:15.848221 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.848047 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-token" pod "d9d0e2c6-68dd-486b-a461-1564218a792b" (UID: "d9d0e2c6-68dd-486b-a461-1564218a792b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:15.862663 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.862635 6466 generic.go:356] "Generic (PLEG): container finished" podID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerID="9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8" exitCode=0 Nov 01 23:01:15.862784 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.862731 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" event={"ID":"d9d0e2c6-68dd-486b-a461-1564218a792b","Type":"ContainerDied","Data":"9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8"} Nov 01 23:01:15.862784 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.862757 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" Nov 01 23:01:15.862886 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.862810 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2" event={"ID":"d9d0e2c6-68dd-486b-a461-1564218a792b","Type":"ContainerDied","Data":"b6512e815daae1b46c48e8795c54098787c65569bde5533e6872463312d5020b"} Nov 01 23:01:15.862886 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.862837 6466 scope.go:122] "RemoveContainer" containerID="9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8" Nov 01 23:01:15.864195 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.864131 6466 generic.go:356] "Generic (PLEG): container finished" podID="45719f2c-6797-450c-8089-e90c493e3b3a" containerID="886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f" exitCode=0 Nov 01 23:01:15.864308 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.864251 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" event={"ID":"45719f2c-6797-450c-8089-e90c493e3b3a","Type":"ContainerDied","Data":"886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f"} Nov 01 23:01:15.864308 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.864259 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" Nov 01 23:01:15.864308 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.864293 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6" event={"ID":"45719f2c-6797-450c-8089-e90c493e3b3a","Type":"ContainerDied","Data":"0b1fe3a4eb740f9ba8a2efeccdeb12ff5a80470fc8a0be2378850e2f4b44f96e"} Nov 01 23:01:15.865867 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.865834 6466 generic.go:356] "Generic (PLEG): container finished" podID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerID="3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5" exitCode=0 Nov 01 23:01:15.865989 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.865895 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" Nov 01 23:01:15.865989 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.865938 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" event={"ID":"7aeac0ad-04ea-4c1b-b273-a0d708cc552f","Type":"ContainerDied","Data":"3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5"} Nov 01 23:01:15.865989 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.865976 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc" event={"ID":"7aeac0ad-04ea-4c1b-b273-a0d708cc552f","Type":"ContainerDied","Data":"c69a3a791e36200ec70051964c81880ad4654f98d7a728a98fa1810d5cab8cb5"} Nov 01 23:01:15.867594 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.867570 6466 generic.go:356] "Generic (PLEG): container finished" podID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerID="74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9" exitCode=0 Nov 01 23:01:15.867721 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.867622 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" event={"ID":"b9169aa9-28c9-4698-8d11-8b0d232d81a0","Type":"ContainerDied","Data":"74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9"} Nov 01 23:01:15.867721 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.867643 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" Nov 01 23:01:15.867721 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.867655 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59" event={"ID":"b9169aa9-28c9-4698-8d11-8b0d232d81a0","Type":"ContainerDied","Data":"388699ed7aa5d39418600c8fc36e94303575ad4a968a31fcccbf68f03b6d39c5"} Nov 01 23:01:15.874506 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.874484 6466 scope.go:122] "RemoveContainer" containerID="9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8" Nov 01 23:01:15.874792 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:01:15.874765 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8\": container with ID starting with 9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8 not found: ID does not exist" containerID="9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8" Nov 01 23:01:15.874894 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.874802 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8"} err="failed to get container status \"9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8\": rpc error: code = NotFound desc = could not find container \"9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8\": container with ID starting with 9be77d1003662afd15aa92f8752e245eba2cee7d82fc1df2bffb454bc75f17b8 not found: ID does not exist" Nov 01 23:01:15.874894 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.874826 6466 scope.go:122] "RemoveContainer" containerID="886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f" Nov 01 23:01:15.885896 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.885879 6466 scope.go:122] "RemoveContainer" containerID="886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f" Nov 01 23:01:15.886193 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:01:15.886174 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f\": container with ID starting with 886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f not found: ID does not exist" containerID="886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f" Nov 01 23:01:15.886250 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.886199 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f"} err="failed to get container status \"886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f\": rpc error: code = NotFound desc = could not find container \"886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f\": container with ID starting with 886cb29eda5099a62af919c030626c484bdca2a97ecb21f1aa50aef338c5718f not found: ID does not exist" Nov 01 23:01:15.886250 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.886216 6466 scope.go:122] "RemoveContainer" containerID="3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5" Nov 01 23:01:15.892616 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.892596 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2"] Nov 01 23:01:15.896061 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.896039 6466 scope.go:122] "RemoveContainer" containerID="3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5" Nov 01 23:01:15.896582 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:01:15.896538 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5\": container with ID starting with 3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5 not found: ID does not exist" containerID="3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5" Nov 01 23:01:15.896582 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.896591 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5"} err="failed to get container status \"3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5\": rpc error: code = NotFound desc = could not find container \"3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5\": container with ID starting with 3fa22f987aee0eb3f93702a34721158fa89764aa1ea960b10a57208745a1d5e5 not found: ID does not exist" Nov 01 23:01:15.896893 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.896616 6466 scope.go:122] "RemoveContainer" containerID="74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9" Nov 01 23:01:15.899833 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.899788 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q5hw2"] Nov 01 23:01:15.905698 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.905682 6466 scope.go:122] "RemoveContainer" containerID="74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9" Nov 01 23:01:15.905963 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:01:15.905945 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9\": container with ID starting with 74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9 not found: ID does not exist" containerID="74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9" Nov 01 23:01:15.906023 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.905969 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9"} err="failed to get container status \"74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9\": rpc error: code = NotFound desc = could not find container \"74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9\": container with ID starting with 74669575cdfe06bd03c750ed38e2fb591d29b3068bd7ea93e8600f8577091fb9 not found: ID does not exist" Nov 01 23:01:15.911317 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.911292 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc"] Nov 01 23:01:15.915269 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.915247 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ggzxc"] Nov 01 23:01:15.927006 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.926931 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59"] Nov 01 23:01:15.933003 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.932965 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9gn59"] Nov 01 23:01:15.944590 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.944566 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6"] Nov 01 23:01:15.947510 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.947487 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.947618 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.947513 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.947618 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.947533 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqtn6\" (UniqueName: \"kubernetes.io/projected/d9d0e2c6-68dd-486b-a461-1564218a792b-kube-api-access-sqtn6\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.947618 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.947547 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9d0e2c6-68dd-486b-a461-1564218a792b-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.950601 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:15.950574 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dgmc6"] Nov 01 23:01:16.339670 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:16.339596 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" path="/var/lib/kubelet/pods/45719f2c-6797-450c-8089-e90c493e3b3a/volumes" Nov 01 23:01:16.340029 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:16.340016 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" path="/var/lib/kubelet/pods/7aeac0ad-04ea-4c1b-b273-a0d708cc552f/volumes" Nov 01 23:01:16.340449 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:16.340436 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" path="/var/lib/kubelet/pods/b9169aa9-28c9-4698-8d11-8b0d232d81a0/volumes" Nov 01 23:01:16.341193 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:16.341179 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" path="/var/lib/kubelet/pods/d9d0e2c6-68dd-486b-a461-1564218a792b/volumes" Nov 01 23:01:19.282545 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:01:19.282510 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:01:20.617401 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.617371 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf"] Nov 01 23:01:20.618197 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618165 6466 cpu_manager.go:414] "containerMap: removing container" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" Nov 01 23:01:20.618197 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618186 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" Nov 01 23:01:20.618381 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618218 6466 cpu_manager.go:414] "containerMap: removing container" podUID="50e404dd-456f-414d-8018-3e8585163874" containerName="revoke-keys" Nov 01 23:01:20.618381 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618227 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e404dd-456f-414d-8018-3e8585163874" containerName="revoke-keys" Nov 01 23:01:20.618381 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618244 6466 cpu_manager.go:414] "containerMap: removing container" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" Nov 01 23:01:20.618381 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618252 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" Nov 01 23:01:20.618381 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618274 6466 cpu_manager.go:414] "containerMap: removing container" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" Nov 01 23:01:20.618381 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618281 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" Nov 01 23:01:20.618381 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618309 6466 cpu_manager.go:414] "containerMap: removing container" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" Nov 01 23:01:20.618381 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618316 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" Nov 01 23:01:20.618784 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618439 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d0e2c6-68dd-486b-a461-1564218a792b" containerName="istio-proxy" Nov 01 23:01:20.618784 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618453 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="45719f2c-6797-450c-8089-e90c493e3b3a" containerName="istio-proxy" Nov 01 23:01:20.618784 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618467 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e404dd-456f-414d-8018-3e8585163874" containerName="revoke-keys" Nov 01 23:01:20.618784 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618479 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9169aa9-28c9-4698-8d11-8b0d232d81a0" containerName="istio-proxy" Nov 01 23:01:20.618784 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.618491 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aeac0ad-04ea-4c1b-b273-a0d708cc552f" containerName="istio-proxy" Nov 01 23:01:20.623621 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.623602 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.631175 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.631149 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-7ae63f-openshift-default-dockercfg-z22v9\"" Nov 01 23:01:20.636227 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.634417 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf"] Nov 01 23:01:20.695194 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.694969 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-socket\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.696288 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.696253 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-credential-socket\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.696439 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.696351 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckmkc\" (UniqueName: \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-kube-api-access-ckmkc\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.696439 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.696403 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-certs\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.696571 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.696474 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99209d5a-dd2a-4999-a77e-942fc8b14aac-istiod-ca-cert\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.696630 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.696576 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-envoy\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.696729 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.696660 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-podinfo\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.696729 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.696693 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-data\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.697086 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.696734 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-token\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.797829 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.797794 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-socket\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798016 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.797861 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-credential-socket\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798016 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.797892 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckmkc\" (UniqueName: \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-kube-api-access-ckmkc\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798016 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.797917 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-certs\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798016 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.797952 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99209d5a-dd2a-4999-a77e-942fc8b14aac-istiod-ca-cert\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798016 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.797988 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-envoy\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798310 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.798033 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-podinfo\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798310 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.798060 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-data\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798310 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.798133 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-token\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798469 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.798324 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-socket\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798469 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.798360 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-credential-socket\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798590 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.798541 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-certs\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798590 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.798565 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-data\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.798845 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.798821 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99209d5a-dd2a-4999-a77e-942fc8b14aac-istiod-ca-cert\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.800491 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.800465 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-envoy\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.800780 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.800756 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-podinfo\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.806502 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.806477 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckmkc\" (UniqueName: \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-kube-api-access-ckmkc\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.806642 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.806624 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-token\") pod \"e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:20.940547 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:20.940514 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:21.222677 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:21.222653 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf"] Nov 01 23:01:21.224853 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:01:21.224820 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99209d5a_dd2a_4999_a77e_942fc8b14aac.slice/crio-8f43e2f0a02fbfcc99d887c489bc65d2f5a3d5ed1c43d854960adc19e5f2e277 WatchSource:0}: Error finding container 8f43e2f0a02fbfcc99d887c489bc65d2f5a3d5ed1c43d854960adc19e5f2e277: Status 404 returned error can't find the container with id 8f43e2f0a02fbfcc99d887c489bc65d2f5a3d5ed1c43d854960adc19e5f2e277 Nov 01 23:01:21.227254 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:21.227218 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:21.227347 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:21.227303 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:21.227389 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:21.227342 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:21.903103 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:21.903065 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" event={"ID":"99209d5a-dd2a-4999-a77e-942fc8b14aac","Type":"ContainerStarted","Data":"fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565"} Nov 01 23:01:21.903556 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:21.903139 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" event={"ID":"99209d5a-dd2a-4999-a77e-942fc8b14aac","Type":"ContainerStarted","Data":"8f43e2f0a02fbfcc99d887c489bc65d2f5a3d5ed1c43d854960adc19e5f2e277"} Nov 01 23:01:21.928821 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:21.928767 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" podStartSLOduration=1.928754015 podStartE2EDuration="1.928754015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:21.926142992 +0000 UTC m=+2285.710581907" watchObservedRunningTime="2026-11-01 23:01:21.928754015 +0000 UTC m=+2285.713192913" Nov 01 23:01:21.941705 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:21.941675 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:22.945981 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:22.945950 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:23.914530 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:23.914496 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:23.916051 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:23.916019 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:25.341545 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.341512 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z"] Nov 01 23:01:25.346835 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.346807 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.357931 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.357904 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:01:25.359051 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.358183 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:01:25.359208 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.358209 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-7ae63f-serving-cert\"" Nov 01 23:01:25.359208 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.358247 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 23:01:25.359208 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.358318 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7x7m\"" Nov 01 23:01:25.360279 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.360256 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z"] Nov 01 23:01:25.436472 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.436438 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5"] Nov 01 23:01:25.440784 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.440760 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.446934 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.446899 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.447093 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.447032 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sdx6\" (UniqueName: \"kubernetes.io/projected/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-kube-api-access-7sdx6\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.447259 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.447238 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.447417 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.447385 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.458035 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.458012 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-flhqn\"" Nov 01 23:01:25.468057 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.468030 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s"] Nov 01 23:01:25.471897 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.471878 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5"] Nov 01 23:01:25.472022 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.471985 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.492398 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.492368 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s"] Nov 01 23:01:25.548818 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.548783 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.548818 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.548819 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmtw\" (UniqueName: \"kubernetes.io/projected/5a202d72-2f71-4a71-aba6-df821c076507-kube-api-access-klmtw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.549029 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.548842 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a202d72-2f71-4a71-aba6-df821c076507-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.549029 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.548924 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.549029 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.548962 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.549029 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549016 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.549226 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549038 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.549226 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549143 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a202d72-2f71-4a71-aba6-df821c076507-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.549226 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549161 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv54c\" (UniqueName: \"kubernetes.io/projected/65c9f274-eb9e-4f06-ae4b-fa127d31c463-kube-api-access-kv54c\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.549226 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549182 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.549226 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549200 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.549226 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549214 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.549614 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549344 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.549614 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549398 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.549614 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549440 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.549614 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549472 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.549614 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549498 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.549614 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549560 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sdx6\" (UniqueName: \"kubernetes.io/projected/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-kube-api-access-7sdx6\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.549614 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549588 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a202d72-2f71-4a71-aba6-df821c076507-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.549614 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549612 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.550067 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549653 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.550067 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.549830 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.550207 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.550156 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.551752 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.551726 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.551894 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.551805 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.558679 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.558657 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sdx6\" (UniqueName: \"kubernetes.io/projected/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-kube-api-access-7sdx6\") pod \"maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.650536 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.650504 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a202d72-2f71-4a71-aba6-df821c076507-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.650742 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.650547 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.650742 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.650564 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.650742 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.650697 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.650939 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.650766 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.650939 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.650794 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmtw\" (UniqueName: \"kubernetes.io/projected/5a202d72-2f71-4a71-aba6-df821c076507-kube-api-access-klmtw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.650939 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.650836 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a202d72-2f71-4a71-aba6-df821c076507-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.650939 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.650870 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.650939 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.650905 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.651223 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.650949 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.651223 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.650990 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.651223 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651019 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.651223 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651037 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a202d72-2f71-4a71-aba6-df821c076507-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.651223 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651051 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.651223 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651063 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv54c\" (UniqueName: \"kubernetes.io/projected/65c9f274-eb9e-4f06-ae4b-fa127d31c463-kube-api-access-kv54c\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.651223 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651124 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.651223 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651137 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.651223 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651154 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.651223 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651212 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.651828 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651268 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.651828 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651314 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.651828 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651382 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a202d72-2f71-4a71-aba6-df821c076507-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.651828 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651414 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.651828 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651708 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.651828 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651707 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.651828 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651711 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.651828 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651732 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.652550 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.651957 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.654102 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.654075 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.654230 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.654171 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a202d72-2f71-4a71-aba6-df821c076507-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.654306 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.654281 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.654499 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.654483 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a202d72-2f71-4a71-aba6-df821c076507-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.659798 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.659772 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c9f274-eb9e-4f06-ae4b-fa127d31c463-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.659903 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.659880 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmtw\" (UniqueName: \"kubernetes.io/projected/5a202d72-2f71-4a71-aba6-df821c076507-kube-api-access-klmtw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.660013 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.659997 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a202d72-2f71-4a71-aba6-df821c076507-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-44l6s\" (UID: \"5a202d72-2f71-4a71-aba6-df821c076507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.660469 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.660449 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv54c\" (UniqueName: \"kubernetes.io/projected/65c9f274-eb9e-4f06-ae4b-fa127d31c463-kube-api-access-kv54c\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wg8g5\" (UID: \"65c9f274-eb9e-4f06-ae4b-fa127d31c463\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.661978 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.661961 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:25.755763 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.755733 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:25.788171 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.788127 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:25.940343 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.940319 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z"] Nov 01 23:01:25.942704 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:01:25.942679 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa2d4fe_9c07_4d99_bd9f_f311eb810a90.slice/crio-e63cca70003fd8e0ca6851d9396a4170c51c59a8c31e84a46c77638676bf257c WatchSource:0}: Error finding container e63cca70003fd8e0ca6851d9396a4170c51c59a8c31e84a46c77638676bf257c: Status 404 returned error can't find the container with id e63cca70003fd8e0ca6851d9396a4170c51c59a8c31e84a46c77638676bf257c Nov 01 23:01:25.944001 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:25.943984 6466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:01:26.042092 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.042061 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5"] Nov 01 23:01:26.044450 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:01:26.044424 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c9f274_eb9e_4f06_ae4b_fa127d31c463.slice/crio-087e393302ffa9c9edba76075c237082067a3beba0938a6ef188ffc1c9ab5f4b WatchSource:0}: Error finding container 087e393302ffa9c9edba76075c237082067a3beba0938a6ef188ffc1c9ab5f4b: Status 404 returned error can't find the container with id 087e393302ffa9c9edba76075c237082067a3beba0938a6ef188ffc1c9ab5f4b Nov 01 23:01:26.047310 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.047278 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:26.047404 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.047346 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:26.047404 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.047375 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:26.095533 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.090067 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s"] Nov 01 23:01:26.095533 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:01:26.092692 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a202d72_2f71_4a71_aba6_df821c076507.slice/crio-a3eda75fe41ecef8ea753742fd553ca9a58d551bbaef8b76c5b6011d3c5aa90b WatchSource:0}: Error finding container a3eda75fe41ecef8ea753742fd553ca9a58d551bbaef8b76c5b6011d3c5aa90b: Status 404 returned error can't find the container with id a3eda75fe41ecef8ea753742fd553ca9a58d551bbaef8b76c5b6011d3c5aa90b Nov 01 23:01:26.095533 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.094252 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:26.095533 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.094316 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:26.095533 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.094344 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:26.931624 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.931080 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" event={"ID":"caa2d4fe-9c07-4d99-bd9f-f311eb810a90","Type":"ContainerStarted","Data":"5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618"} Nov 01 23:01:26.931624 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.931154 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" event={"ID":"caa2d4fe-9c07-4d99-bd9f-f311eb810a90","Type":"ContainerStarted","Data":"e63cca70003fd8e0ca6851d9396a4170c51c59a8c31e84a46c77638676bf257c"} Nov 01 23:01:26.931624 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.931290 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:26.932754 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.932728 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" event={"ID":"5a202d72-2f71-4a71-aba6-df821c076507","Type":"ContainerStarted","Data":"0db55acefe0db122cedc6de3502943f5643c02ed2bb1db83b22485f2795679f3"} Nov 01 23:01:26.932822 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.932771 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" event={"ID":"5a202d72-2f71-4a71-aba6-df821c076507","Type":"ContainerStarted","Data":"a3eda75fe41ecef8ea753742fd553ca9a58d551bbaef8b76c5b6011d3c5aa90b"} Nov 01 23:01:26.934871 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.934851 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" event={"ID":"65c9f274-eb9e-4f06-ae4b-fa127d31c463","Type":"ContainerStarted","Data":"9b38b41991414c58de0017e3377307e30cf29ccc88bce2fd01eb52f8ab7c08c5"} Nov 01 23:01:26.934915 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:26.934895 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" event={"ID":"65c9f274-eb9e-4f06-ae4b-fa127d31c463","Type":"ContainerStarted","Data":"087e393302ffa9c9edba76075c237082067a3beba0938a6ef188ffc1c9ab5f4b"} Nov 01 23:01:27.010534 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:27.008422 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" podStartSLOduration=1.5774869900000001 podStartE2EDuration="2.008403786s" totalImagesPullingTime="430.916796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:26.981018516 +0000 UTC m=+2290.765457441" watchObservedRunningTime="2026-11-01 23:01:27.008403786 +0000 UTC m=+2290.792842710" Nov 01 23:01:27.010534 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:27.008711 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" podStartSLOduration=2.008705614 podStartE2EDuration="2.008705614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:27.007798886 +0000 UTC m=+2290.792237822" watchObservedRunningTime="2026-11-01 23:01:27.008705614 +0000 UTC m=+2290.793144536" Nov 01 23:01:27.064130 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:27.063037 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" podStartSLOduration=2.063016995 podStartE2EDuration="2.063016995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:27.060694572 +0000 UTC m=+2290.845133506" watchObservedRunningTime="2026-11-01 23:01:27.063016995 +0000 UTC m=+2290.847455917" Nov 01 23:01:27.757131 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:27.756482 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:27.765134 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:27.761596 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wg8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 23:01:27.765134 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:27.761656 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" podUID="65c9f274-eb9e-4f06-ae4b-fa127d31c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 23:01:27.790309 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:27.790263 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:27.792318 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:27.792283 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-44l6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 23:01:27.792475 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:27.792356 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" podUID="5a202d72-2f71-4a71-aba6-df821c076507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 23:01:28.769296 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:28.769252 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wg8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 23:01:28.769928 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:28.769893 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" podUID="65c9f274-eb9e-4f06-ae4b-fa127d31c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 23:01:28.789241 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:28.789200 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-44l6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 23:01:28.789425 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:28.789273 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" podUID="5a202d72-2f71-4a71-aba6-df821c076507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 23:01:29.759131 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:29.758140 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wg8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 23:01:29.759131 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:29.758295 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" podUID="65c9f274-eb9e-4f06-ae4b-fa127d31c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 23:01:29.790130 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:29.789513 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-44l6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 23:01:29.790130 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:29.789576 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" podUID="5a202d72-2f71-4a71-aba6-df821c076507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 23:01:30.757132 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:30.756503 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wg8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 23:01:30.757132 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:30.756570 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" podUID="65c9f274-eb9e-4f06-ae4b-fa127d31c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 23:01:30.804132 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:30.803500 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-44l6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 23:01:30.804132 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:30.803562 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" podUID="5a202d72-2f71-4a71-aba6-df821c076507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 23:01:31.759132 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:31.758508 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wg8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 23:01:31.759132 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:31.758576 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" podUID="65c9f274-eb9e-4f06-ae4b-fa127d31c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 23:01:31.790132 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:31.789429 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-44l6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 23:01:31.790132 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:31.789506 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" podUID="5a202d72-2f71-4a71-aba6-df821c076507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 23:01:32.760131 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:32.759555 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wg8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 23:01:32.760131 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:32.759629 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" podUID="65c9f274-eb9e-4f06-ae4b-fa127d31c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 23:01:32.788642 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:32.788608 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-44l6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 23:01:32.788824 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:32.788660 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" podUID="5a202d72-2f71-4a71-aba6-df821c076507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 23:01:32.975344 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:32.975314 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:01:33.759571 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:33.759529 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wg8g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 01 23:01:33.759871 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:33.759842 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" podUID="65c9f274-eb9e-4f06-ae4b-fa127d31c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 01 23:01:33.788994 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:33.788960 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-44l6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 01 23:01:33.789558 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:33.789007 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" podUID="5a202d72-2f71-4a71-aba6-df821c076507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 01 23:01:35.255214 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:35.253701 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:35.255214 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:35.255036 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:35.256139 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:35.256085 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" Nov 01 23:01:35.370503 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:35.368724 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:35.370503 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:35.370011 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:35.370955 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:35.370940 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-44l6s" Nov 01 23:01:38.915953 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:38.915919 6466 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 23:01:38.916421 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:38.915973 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 23:01:39.067399 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:39.067366 6466 generic.go:356] "Generic (PLEG): container finished" podID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerID="fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565" exitCode=0 Nov 01 23:01:39.067592 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:39.067431 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" event={"ID":"99209d5a-dd2a-4999-a77e-942fc8b14aac","Type":"ContainerDied","Data":"fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565"} Nov 01 23:01:39.067842 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:39.067811 6466 scope.go:122] "RemoveContainer" containerID="fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565" Nov 01 23:01:39.069232 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:39.069202 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:39.069302 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:39.069275 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:39.069339 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:39.069324 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:01:40.075573 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:40.075537 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" event={"ID":"99209d5a-dd2a-4999-a77e-942fc8b14aac","Type":"ContainerStarted","Data":"076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185"} Nov 01 23:01:40.958131 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:40.955411 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:40.961154 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:40.960228 6466 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 23:01:40.961154 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:40.960286 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 23:01:41.957131 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:41.956485 6466 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 23:01:41.957131 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:41.956547 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 23:01:42.954131 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:42.952452 6466 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 23:01:42.954131 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:42.952514 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 23:01:43.950573 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:43.950529 6466 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 23:01:43.951248 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:43.951212 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 23:01:44.943531 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:44.943491 6466 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 23:01:44.943877 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:44.943844 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 23:01:45.944881 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:45.944852 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:45.945349 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:45.945265 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:45.946042 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:01:45.946020 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:01:49.287352 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:01:49.287312 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:02:19.280019 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:02:19.279978 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:02:33.627532 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:33.627452 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z"] Nov 01 23:02:33.627964 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:33.627716 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" podUID="caa2d4fe-9c07-4d99-bd9f-f311eb810a90" containerName="maas-api" containerID="cri-o://5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618" gracePeriod=30 Nov 01 23:02:33.904883 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:33.904860 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:02:34.071560 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.071530 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-metrics-certs\") pod \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " Nov 01 23:02:34.071764 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.071575 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-openshift-service-ca\") pod \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " Nov 01 23:02:34.071764 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.071599 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-maas-api-tls\") pod \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " Nov 01 23:02:34.071764 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.071665 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-kube-api-access-7sdx6\" (UniqueName: \"kubernetes.io/projected/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-kube-api-access-7sdx6\") pod \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\" (UID: \"caa2d4fe-9c07-4d99-bd9f-f311eb810a90\") " Nov 01 23:02:34.071968 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.071923 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-openshift-service-ca" pod "caa2d4fe-9c07-4d99-bd9f-f311eb810a90" (UID: "caa2d4fe-9c07-4d99-bd9f-f311eb810a90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:34.073865 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.073839 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-metrics-certs" pod "caa2d4fe-9c07-4d99-bd9f-f311eb810a90" (UID: "caa2d4fe-9c07-4d99-bd9f-f311eb810a90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:34.073865 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.073858 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-maas-api-tls" pod "caa2d4fe-9c07-4d99-bd9f-f311eb810a90" (UID: "caa2d4fe-9c07-4d99-bd9f-f311eb810a90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:34.074047 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.073839 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-kube-api-access-7sdx6" pod "caa2d4fe-9c07-4d99-bd9f-f311eb810a90" (UID: "caa2d4fe-9c07-4d99-bd9f-f311eb810a90"). InnerVolumeSpecName "kube-api-access-7sdx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:34.172636 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.172595 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:34.172636 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.172627 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sdx6\" (UniqueName: \"kubernetes.io/projected/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-kube-api-access-7sdx6\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:34.172636 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.172640 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:34.172899 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.172650 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa2d4fe-9c07-4d99-bd9f-f311eb810a90-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:34.364013 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.363982 6466 generic.go:356] "Generic (PLEG): container finished" podID="caa2d4fe-9c07-4d99-bd9f-f311eb810a90" containerID="5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618" exitCode=0 Nov 01 23:02:34.364227 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.364037 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" Nov 01 23:02:34.364227 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.364060 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" event={"ID":"caa2d4fe-9c07-4d99-bd9f-f311eb810a90","Type":"ContainerDied","Data":"5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618"} Nov 01 23:02:34.364227 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.364129 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z" event={"ID":"caa2d4fe-9c07-4d99-bd9f-f311eb810a90","Type":"ContainerDied","Data":"e63cca70003fd8e0ca6851d9396a4170c51c59a8c31e84a46c77638676bf257c"} Nov 01 23:02:34.364227 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.364153 6466 scope.go:122] "RemoveContainer" containerID="5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618" Nov 01 23:02:34.373977 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.373956 6466 scope.go:122] "RemoveContainer" containerID="5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618" Nov 01 23:02:34.374368 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:02:34.374336 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618\": container with ID starting with 5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618 not found: ID does not exist" containerID="5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618" Nov 01 23:02:34.374422 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.374376 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618"} err="failed to get container status \"5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618\": rpc error: code = NotFound desc = could not find container \"5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618\": container with ID starting with 5065f4ac20d108d76733b6c62721c7ba70e9b83ea58f46d631c6d79c3746a618 not found: ID does not exist" Nov 01 23:02:34.381826 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.381802 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z"] Nov 01 23:02:34.385470 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:34.385448 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7ae63f-556866bc7b-k2h9z"] Nov 01 23:02:36.341547 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:36.341273 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa2d4fe-9c07-4d99-bd9f-f311eb810a90" path="/var/lib/kubelet/pods/caa2d4fe-9c07-4d99-bd9f-f311eb810a90/volumes" Nov 01 23:02:40.895617 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:40.895573 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf"] Nov 01 23:02:40.896136 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:40.895841 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" containerID="cri-o://076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185" gracePeriod=30 Nov 01 23:02:45.947010 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:45.946972 6466 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 01 23:02:45.947409 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:45.947041 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 01 23:02:46.140848 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.140814 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:02:46.198310 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198233 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99209d5a-dd2a-4999-a77e-942fc8b14aac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99209d5a-dd2a-4999-a77e-942fc8b14aac-istiod-ca-cert\") pod \"99209d5a-dd2a-4999-a77e-942fc8b14aac\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " Nov 01 23:02:46.198468 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198365 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-token\" (UniqueName: \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-token\") pod \"99209d5a-dd2a-4999-a77e-942fc8b14aac\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " Nov 01 23:02:46.198468 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198426 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-envoy\") pod \"99209d5a-dd2a-4999-a77e-942fc8b14aac\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " Nov 01 23:02:46.198597 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198468 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-socket\") pod \"99209d5a-dd2a-4999-a77e-942fc8b14aac\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " Nov 01 23:02:46.198597 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198505 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-kube-api-access-ckmkc\" (UniqueName: \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-kube-api-access-ckmkc\") pod \"99209d5a-dd2a-4999-a77e-942fc8b14aac\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " Nov 01 23:02:46.198597 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198535 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-certs\") pod \"99209d5a-dd2a-4999-a77e-942fc8b14aac\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " Nov 01 23:02:46.198597 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198564 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-credential-socket\") pod \"99209d5a-dd2a-4999-a77e-942fc8b14aac\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " Nov 01 23:02:46.198811 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198622 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99209d5a-dd2a-4999-a77e-942fc8b14aac-istiod-ca-cert" pod "99209d5a-dd2a-4999-a77e-942fc8b14aac" (UID: "99209d5a-dd2a-4999-a77e-942fc8b14aac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:46.198811 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198681 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-data\") pod \"99209d5a-dd2a-4999-a77e-942fc8b14aac\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " Nov 01 23:02:46.198936 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198808 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-socket" pod "99209d5a-dd2a-4999-a77e-942fc8b14aac" (UID: "99209d5a-dd2a-4999-a77e-942fc8b14aac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:46.198936 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198819 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-credential-socket" pod "99209d5a-dd2a-4999-a77e-942fc8b14aac" (UID: "99209d5a-dd2a-4999-a77e-942fc8b14aac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:46.198936 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198850 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-certs" pod "99209d5a-dd2a-4999-a77e-942fc8b14aac" (UID: "99209d5a-dd2a-4999-a77e-942fc8b14aac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:46.198936 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.198869 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-podinfo\") pod \"99209d5a-dd2a-4999-a77e-942fc8b14aac\" (UID: \"99209d5a-dd2a-4999-a77e-942fc8b14aac\") " Nov 01 23:02:46.199186 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.199023 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-data" pod "99209d5a-dd2a-4999-a77e-942fc8b14aac" (UID: "99209d5a-dd2a-4999-a77e-942fc8b14aac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:46.199307 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.199286 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:46.199375 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.199311 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:46.199375 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.199326 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:46.199375 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.199339 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:46.199375 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.199353 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99209d5a-dd2a-4999-a77e-942fc8b14aac-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:46.200648 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.200599 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-token" pod "99209d5a-dd2a-4999-a77e-942fc8b14aac" (UID: "99209d5a-dd2a-4999-a77e-942fc8b14aac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:46.200781 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.200681 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-envoy" pod "99209d5a-dd2a-4999-a77e-942fc8b14aac" (UID: "99209d5a-dd2a-4999-a77e-942fc8b14aac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:46.200781 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.200682 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-kube-api-access-ckmkc" pod "99209d5a-dd2a-4999-a77e-942fc8b14aac" (UID: "99209d5a-dd2a-4999-a77e-942fc8b14aac"). InnerVolumeSpecName "kube-api-access-ckmkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:46.200866 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.200825 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-podinfo" pod "99209d5a-dd2a-4999-a77e-942fc8b14aac" (UID: "99209d5a-dd2a-4999-a77e-942fc8b14aac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:02:46.300646 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.300614 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:46.300646 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.300639 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckmkc\" (UniqueName: \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-kube-api-access-ckmkc\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:46.300646 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.300649 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:46.300899 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.300659 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99209d5a-dd2a-4999-a77e-942fc8b14aac-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:46.423350 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.423317 6466 generic.go:356] "Generic (PLEG): container finished" podID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerID="076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185" exitCode=0 Nov 01 23:02:46.423556 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.423364 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" Nov 01 23:02:46.423556 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.423374 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" event={"ID":"99209d5a-dd2a-4999-a77e-942fc8b14aac","Type":"ContainerDied","Data":"076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185"} Nov 01 23:02:46.423556 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.423425 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf" event={"ID":"99209d5a-dd2a-4999-a77e-942fc8b14aac","Type":"ContainerDied","Data":"8f43e2f0a02fbfcc99d887c489bc65d2f5a3d5ed1c43d854960adc19e5f2e277"} Nov 01 23:02:46.423556 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.423455 6466 scope.go:122] "RemoveContainer" containerID="076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185" Nov 01 23:02:46.432412 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.432390 6466 scope.go:122] "RemoveContainer" containerID="fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565" Nov 01 23:02:46.440486 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.440463 6466 scope.go:122] "RemoveContainer" containerID="076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185" Nov 01 23:02:46.440846 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:02:46.440819 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185\": container with ID starting with 076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185 not found: ID does not exist" containerID="076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185" Nov 01 23:02:46.440921 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.440853 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185"} err="failed to get container status \"076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185\": rpc error: code = NotFound desc = could not find container \"076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185\": container with ID starting with 076a6bd1a50190f0be25328511e645d6c5255f7e22b0189651096d81fbe74185 not found: ID does not exist" Nov 01 23:02:46.440921 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.440870 6466 scope.go:122] "RemoveContainer" containerID="fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565" Nov 01 23:02:46.441156 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:02:46.441135 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565\": container with ID starting with fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565 not found: ID does not exist" containerID="fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565" Nov 01 23:02:46.441204 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.441165 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565"} err="failed to get container status \"fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565\": rpc error: code = NotFound desc = could not find container \"fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565\": container with ID starting with fff80796e551afa2d5d4dbac440870e86a18036f1b8d87ea41190d16d49be565 not found: ID does not exist" Nov 01 23:02:46.441335 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.441320 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf"] Nov 01 23:02:46.444893 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:46.444873 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ae63f-openshift-default-7459f7d8b52h9pf"] Nov 01 23:02:48.339281 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:02:48.339246 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" path="/var/lib/kubelet/pods/99209d5a-dd2a-4999-a77e-942fc8b14aac/volumes" Nov 01 23:02:49.280088 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:02:49.280051 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:03:10.596005 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.595925 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw"] Nov 01 23:03:10.596526 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.596476 6466 cpu_manager.go:414] "containerMap: removing container" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" Nov 01 23:03:10.596526 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.596494 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" Nov 01 23:03:10.596526 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.596509 6466 cpu_manager.go:414] "containerMap: removing container" podUID="caa2d4fe-9c07-4d99-bd9f-f311eb810a90" containerName="maas-api" Nov 01 23:03:10.596526 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.596516 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa2d4fe-9c07-4d99-bd9f-f311eb810a90" containerName="maas-api" Nov 01 23:03:10.596722 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.596547 6466 cpu_manager.go:414] "containerMap: removing container" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" Nov 01 23:03:10.596722 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.596554 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" Nov 01 23:03:10.596722 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.596663 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" Nov 01 23:03:10.596722 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.596674 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa2d4fe-9c07-4d99-bd9f-f311eb810a90" containerName="maas-api" Nov 01 23:03:10.596722 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.596688 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="99209d5a-dd2a-4999-a77e-942fc8b14aac" containerName="istio-proxy" Nov 01 23:03:10.601926 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.601898 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.621219 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.621189 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw"] Nov 01 23:03:10.755665 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.755625 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.755882 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.755674 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06103ca9-e37d-4286-960e-cd1c06da76af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.755882 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.755810 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.755978 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.755886 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkghn\" (UniqueName: \"kubernetes.io/projected/06103ca9-e37d-4286-960e-cd1c06da76af-kube-api-access-mkghn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.755978 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.755938 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.755978 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.755961 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06103ca9-e37d-4286-960e-cd1c06da76af-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.756098 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.755985 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.756098 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.756069 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.756226 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.756097 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06103ca9-e37d-4286-960e-cd1c06da76af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.857632 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.857522 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.857632 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.857588 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkghn\" (UniqueName: \"kubernetes.io/projected/06103ca9-e37d-4286-960e-cd1c06da76af-kube-api-access-mkghn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.857879 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.857647 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.857879 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.857673 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06103ca9-e37d-4286-960e-cd1c06da76af-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.857879 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.857699 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.857879 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.857732 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.858221 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.857862 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06103ca9-e37d-4286-960e-cd1c06da76af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.858221 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.857962 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.858221 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.857984 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.858221 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.858014 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06103ca9-e37d-4286-960e-cd1c06da76af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.858221 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.858027 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.858221 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.858052 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.858447 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.858313 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.858580 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.858559 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06103ca9-e37d-4286-960e-cd1c06da76af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.860041 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.860013 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06103ca9-e37d-4286-960e-cd1c06da76af-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.860329 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.860311 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06103ca9-e37d-4286-960e-cd1c06da76af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.867556 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.867532 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06103ca9-e37d-4286-960e-cd1c06da76af-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.867860 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.867840 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkghn\" (UniqueName: \"kubernetes.io/projected/06103ca9-e37d-4286-960e-cd1c06da76af-kube-api-access-mkghn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-wfrzw\" (UID: \"06103ca9-e37d-4286-960e-cd1c06da76af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:10.915722 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:10.915689 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:11.193385 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:11.193344 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw"] Nov 01 23:03:11.196862 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:03:11.196834 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06103ca9_e37d_4286_960e_cd1c06da76af.slice/crio-cbaa22bd289feb959227726e28f1bed61ef24d24995ba00975eb4c2c415ea43a WatchSource:0}: Error finding container cbaa22bd289feb959227726e28f1bed61ef24d24995ba00975eb4c2c415ea43a: Status 404 returned error can't find the container with id cbaa22bd289feb959227726e28f1bed61ef24d24995ba00975eb4c2c415ea43a Nov 01 23:03:11.199612 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:11.199574 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:03:11.199721 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:11.199652 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:03:11.199721 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:11.199700 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:03:11.556984 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:11.556897 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" event={"ID":"06103ca9-e37d-4286-960e-cd1c06da76af","Type":"ContainerStarted","Data":"c3c05a0a077f575e0fc922102e115f359a8a5834b76f197c342f6013972bc34d"} Nov 01 23:03:11.556984 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:11.556956 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" event={"ID":"06103ca9-e37d-4286-960e-cd1c06da76af","Type":"ContainerStarted","Data":"cbaa22bd289feb959227726e28f1bed61ef24d24995ba00975eb4c2c415ea43a"} Nov 01 23:03:11.594686 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:11.594628 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" podStartSLOduration=1.594614218 podStartE2EDuration="1.594614218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:11.59266433 +0000 UTC m=+2395.377103250" watchObservedRunningTime="2026-11-01 23:03:11.594614218 +0000 UTC m=+2395.379053139" Nov 01 23:03:11.916439 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:11.916405 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:12.921426 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:12.921396 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:13.566781 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:13.566753 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:13.567893 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:03:13.567874 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:03:19.281261 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:03:19.281222 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:03:49.280541 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:03:49.280499 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:04:19.280296 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:04:19.280257 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:04:40.020579 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:40.020541 6466 generic.go:356] "Generic (PLEG): container finished" podID="06103ca9-e37d-4286-960e-cd1c06da76af" containerID="c3c05a0a077f575e0fc922102e115f359a8a5834b76f197c342f6013972bc34d" exitCode=0 Nov 01 23:04:40.021025 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:40.020620 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" event={"ID":"06103ca9-e37d-4286-960e-cd1c06da76af","Type":"ContainerDied","Data":"c3c05a0a077f575e0fc922102e115f359a8a5834b76f197c342f6013972bc34d"} Nov 01 23:04:40.021096 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:40.021077 6466 scope.go:122] "RemoveContainer" containerID="c3c05a0a077f575e0fc922102e115f359a8a5834b76f197c342f6013972bc34d" Nov 01 23:04:40.022356 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:40.022327 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:04:40.022422 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:40.022396 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:04:40.022461 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:40.022430 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:04:41.026988 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:41.026952 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" event={"ID":"06103ca9-e37d-4286-960e-cd1c06da76af","Type":"ContainerStarted","Data":"b094c35017b956042cdadff4d881a154bbb514a308656d8f9875076a17f37740"} Nov 01 23:04:41.916370 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:41.916331 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:04:41.917424 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:41.917396 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wfrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:04:41.917579 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:41.917452 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" podUID="06103ca9-e37d-4286-960e-cd1c06da76af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 23:04:42.916134 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:42.916071 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wfrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:04:42.916541 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:42.916147 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" podUID="06103ca9-e37d-4286-960e-cd1c06da76af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 23:04:43.567414 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:43.567380 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:04:43.916919 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:43.916885 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wfrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:04:43.917331 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:43.916948 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" podUID="06103ca9-e37d-4286-960e-cd1c06da76af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 23:04:44.916581 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:44.916544 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wfrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:04:44.916790 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:44.916609 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" podUID="06103ca9-e37d-4286-960e-cd1c06da76af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 23:04:45.916616 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:45.916579 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wfrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:04:45.917049 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:45.916632 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" podUID="06103ca9-e37d-4286-960e-cd1c06da76af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 23:04:46.916389 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:46.916341 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wfrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 01 23:04:46.916584 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:46.916426 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" podUID="06103ca9-e37d-4286-960e-cd1c06da76af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 01 23:04:48.916695 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:48.916644 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wfrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:48.917171 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:48.916735 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" podUID="06103ca9-e37d-4286-960e-cd1c06da76af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:49.312565 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:04:49.312454 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:04:49.960269 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:49.960228 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wfrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:49.960900 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:49.960867 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" podUID="06103ca9-e37d-4286-960e-cd1c06da76af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:50.961944 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:50.961902 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wfrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:50.962480 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:50.961981 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" podUID="06103ca9-e37d-4286-960e-cd1c06da76af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:52.003274 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:52.003235 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wfrzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:52.003679 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:52.003288 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" podUID="06103ca9-e37d-4286-960e-cd1c06da76af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:52.879039 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:52.879011 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:04:52.879947 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:04:52.879930 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wfrzw" Nov 01 23:05:11.651644 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.651604 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm"] Nov 01 23:05:11.657772 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.657747 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.674206 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.674175 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm"] Nov 01 23:05:11.676730 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.676710 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:05:11.677064 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.677003 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:05:11.677191 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.677028 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ee0082-serving-cert\"" Nov 01 23:05:11.678020 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.678000 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 23:05:11.678241 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.678179 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7x7m\"" Nov 01 23:05:11.686804 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.686775 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg"] Nov 01 23:05:11.690737 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.690714 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:05:11.705309 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.705276 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a291003d-dockercfg-7vm7q\"" Nov 01 23:05:11.705627 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.705602 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg"] Nov 01 23:05:11.770504 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.770473 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ee0082\"" Nov 01 23:05:11.801257 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.801225 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9232acb1-beac-445f-a0ab-5f9280b06116-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg\" (UID: \"9232acb1-beac-445f-a0ab-5f9280b06116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:05:11.801257 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.801265 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.801485 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.801410 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.801485 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.801456 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9qtb\" (UniqueName: \"kubernetes.io/projected/9232acb1-beac-445f-a0ab-5f9280b06116-kube-api-access-v9qtb\") pod \"payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg\" (UID: \"9232acb1-beac-445f-a0ab-5f9280b06116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:05:11.801579 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.801559 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d9sd\" (UniqueName: \"kubernetes.io/projected/605265bf-8f1a-4910-9d04-d5828c399405-kube-api-access-8d9sd\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.801620 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.801596 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605265bf-8f1a-4910-9d04-d5828c399405-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.903004 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.902898 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d9sd\" (UniqueName: \"kubernetes.io/projected/605265bf-8f1a-4910-9d04-d5828c399405-kube-api-access-8d9sd\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.903749 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.903708 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605265bf-8f1a-4910-9d04-d5828c399405-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.904029 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.904000 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9232acb1-beac-445f-a0ab-5f9280b06116-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg\" (UID: \"9232acb1-beac-445f-a0ab-5f9280b06116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:05:11.904175 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.904046 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.904250 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.904191 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.904250 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.904234 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qtb\" (UniqueName: \"kubernetes.io/projected/9232acb1-beac-445f-a0ab-5f9280b06116-kube-api-access-v9qtb\") pod \"payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg\" (UID: \"9232acb1-beac-445f-a0ab-5f9280b06116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:05:11.910543 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.910499 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605265bf-8f1a-4910-9d04-d5828c399405-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.911305 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.911278 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9232acb1-beac-445f-a0ab-5f9280b06116-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg\" (UID: \"9232acb1-beac-445f-a0ab-5f9280b06116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:05:11.912955 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.912929 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d9sd\" (UniqueName: \"kubernetes.io/projected/605265bf-8f1a-4910-9d04-d5828c399405-kube-api-access-8d9sd\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.914792 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.914765 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.919276 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.919253 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:11.921290 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.921262 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qtb\" (UniqueName: \"kubernetes.io/projected/9232acb1-beac-445f-a0ab-5f9280b06116-kube-api-access-v9qtb\") pod \"payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg\" (UID: \"9232acb1-beac-445f-a0ab-5f9280b06116\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:05:11.976316 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:11.976272 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:12.013518 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:12.013473 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:05:12.264406 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:12.264386 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm"] Nov 01 23:05:12.266918 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:05:12.266887 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605265bf_8f1a_4910_9d04_d5828c399405.slice/crio-260a807b8b61f8c48509b5368e9f4f3b4a7af3fecbe09ccc08386bc09556b137 WatchSource:0}: Error finding container 260a807b8b61f8c48509b5368e9f4f3b4a7af3fecbe09ccc08386bc09556b137: Status 404 returned error can't find the container with id 260a807b8b61f8c48509b5368e9f4f3b4a7af3fecbe09ccc08386bc09556b137 Nov 01 23:05:12.316468 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:12.316431 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg"] Nov 01 23:05:12.321331 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:05:12.321302 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9232acb1_beac_445f_a0ab_5f9280b06116.slice/crio-aebf2a4b7658c8dcc6c4eb806d62c7501440275e2a823ed0ac5cb06ae4537391 WatchSource:0}: Error finding container aebf2a4b7658c8dcc6c4eb806d62c7501440275e2a823ed0ac5cb06ae4537391: Status 404 returned error can't find the container with id aebf2a4b7658c8dcc6c4eb806d62c7501440275e2a823ed0ac5cb06ae4537391 Nov 01 23:05:13.231519 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:13.231476 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" event={"ID":"9232acb1-beac-445f-a0ab-5f9280b06116","Type":"ContainerStarted","Data":"24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d"} Nov 01 23:05:13.232003 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:13.231574 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" event={"ID":"9232acb1-beac-445f-a0ab-5f9280b06116","Type":"ContainerStarted","Data":"aebf2a4b7658c8dcc6c4eb806d62c7501440275e2a823ed0ac5cb06ae4537391"} Nov 01 23:05:13.232003 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:13.231608 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:05:13.233462 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:13.233428 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" event={"ID":"605265bf-8f1a-4910-9d04-d5828c399405","Type":"ContainerStarted","Data":"ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849"} Nov 01 23:05:13.233596 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:13.233484 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" event={"ID":"605265bf-8f1a-4910-9d04-d5828c399405","Type":"ContainerStarted","Data":"260a807b8b61f8c48509b5368e9f4f3b4a7af3fecbe09ccc08386bc09556b137"} Nov 01 23:05:13.233681 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:13.233665 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:13.255460 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:13.255403 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" podStartSLOduration=2.255385743 podStartE2EDuration="2.255385743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:13.251186965 +0000 UTC m=+2517.035625886" watchObservedRunningTime="2026-11-01 23:05:13.255385743 +0000 UTC m=+2517.039824672" Nov 01 23:05:13.274550 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:13.274500 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" podStartSLOduration=1.752601997 podStartE2EDuration="2.274486841s" totalImagesPullingTime="521.884844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:13.272918834 +0000 UTC m=+2517.057357755" watchObservedRunningTime="2026-11-01 23:05:13.274486841 +0000 UTC m=+2517.058925762" Nov 01 23:05:19.243980 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:19.243950 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:05:19.290358 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:05:19.290315 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:05:24.241190 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:05:24.241157 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:05:49.280930 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:05:49.280890 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:06:18.347337 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:18.347296 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg"] Nov 01 23:06:18.831843 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:18.831806 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm"] Nov 01 23:06:18.832060 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:18.832033 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" podUID="605265bf-8f1a-4910-9d04-d5828c399405" containerName="maas-api" containerID="cri-o://ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849" gracePeriod=30 Nov 01 23:06:19.119568 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.119543 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:06:19.247324 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.247288 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-metrics-certs\" (UniqueName: \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-metrics-certs\") pod \"605265bf-8f1a-4910-9d04-d5828c399405\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " Nov 01 23:06:19.247590 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.247575 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-maas-api-tls\") pod \"605265bf-8f1a-4910-9d04-d5828c399405\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " Nov 01 23:06:19.247736 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.247723 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605265bf-8f1a-4910-9d04-d5828c399405-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605265bf-8f1a-4910-9d04-d5828c399405-openshift-service-ca\") pod \"605265bf-8f1a-4910-9d04-d5828c399405\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " Nov 01 23:06:19.247907 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.247894 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605265bf-8f1a-4910-9d04-d5828c399405-kube-api-access-8d9sd\" (UniqueName: \"kubernetes.io/projected/605265bf-8f1a-4910-9d04-d5828c399405-kube-api-access-8d9sd\") pod \"605265bf-8f1a-4910-9d04-d5828c399405\" (UID: \"605265bf-8f1a-4910-9d04-d5828c399405\") " Nov 01 23:06:19.249410 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.249380 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605265bf-8f1a-4910-9d04-d5828c399405-openshift-service-ca" pod "605265bf-8f1a-4910-9d04-d5828c399405" (UID: "605265bf-8f1a-4910-9d04-d5828c399405"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:19.254148 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.253449 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605265bf-8f1a-4910-9d04-d5828c399405-kube-api-access-8d9sd" pod "605265bf-8f1a-4910-9d04-d5828c399405" (UID: "605265bf-8f1a-4910-9d04-d5828c399405"). InnerVolumeSpecName "kube-api-access-8d9sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:19.259345 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.258437 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-metrics-certs" pod "605265bf-8f1a-4910-9d04-d5828c399405" (UID: "605265bf-8f1a-4910-9d04-d5828c399405"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:19.263326 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.263285 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-maas-api-tls" pod "605265bf-8f1a-4910-9d04-d5828c399405" (UID: "605265bf-8f1a-4910-9d04-d5828c399405"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:19.307573 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:06:19.307536 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:06:19.349474 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.349381 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:19.349474 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.349414 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605265bf-8f1a-4910-9d04-d5828c399405-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:19.349474 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.349431 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605265bf-8f1a-4910-9d04-d5828c399405-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:19.349474 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.349446 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d9sd\" (UniqueName: \"kubernetes.io/projected/605265bf-8f1a-4910-9d04-d5828c399405-kube-api-access-8d9sd\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:19.616301 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.616217 6466 generic.go:356] "Generic (PLEG): container finished" podID="605265bf-8f1a-4910-9d04-d5828c399405" containerID="ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849" exitCode=0 Nov 01 23:06:19.616301 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.616264 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" Nov 01 23:06:19.616497 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.616301 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" event={"ID":"605265bf-8f1a-4910-9d04-d5828c399405","Type":"ContainerDied","Data":"ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849"} Nov 01 23:06:19.616497 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.616363 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm" event={"ID":"605265bf-8f1a-4910-9d04-d5828c399405","Type":"ContainerDied","Data":"260a807b8b61f8c48509b5368e9f4f3b4a7af3fecbe09ccc08386bc09556b137"} Nov 01 23:06:19.616497 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.616390 6466 scope.go:122] "RemoveContainer" containerID="ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849" Nov 01 23:06:19.634956 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.634930 6466 scope.go:122] "RemoveContainer" containerID="ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849" Nov 01 23:06:19.635309 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:06:19.635265 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849\": container with ID starting with ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849 not found: ID does not exist" containerID="ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849" Nov 01 23:06:19.635452 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.635312 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849"} err="failed to get container status \"ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849\": rpc error: code = NotFound desc = could not find container \"ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849\": container with ID starting with ff549206cb329ee9f1f62ae073f3c8c940ec1a342bfe643a931ca8fcd5416849 not found: ID does not exist" Nov 01 23:06:19.645890 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.645864 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm"] Nov 01 23:06:19.651571 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:19.651544 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee0082-5d69bcdc94-672cm"] Nov 01 23:06:20.339946 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:20.339912 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605265bf-8f1a-4910-9d04-d5828c399405" path="/var/lib/kubelet/pods/605265bf-8f1a-4910-9d04-d5828c399405/volumes" Nov 01 23:06:23.348102 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:23.348044 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" podUID="9232acb1-beac-445f-a0ab-5f9280b06116" containerName="payload-pre-processing" containerID="cri-o://24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d" gracePeriod=25 Nov 01 23:06:24.239410 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:24.239366 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" podUID="9232acb1-beac-445f-a0ab-5f9280b06116" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:06:34.258133 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:34.256202 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" podUID="9232acb1-beac-445f-a0ab-5f9280b06116" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:06:43.598856 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.598830 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:06:43.711570 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.711533 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9232acb1-beac-445f-a0ab-5f9280b06116-kube-api-access-v9qtb\" (UniqueName: \"kubernetes.io/projected/9232acb1-beac-445f-a0ab-5f9280b06116-kube-api-access-v9qtb\") pod \"9232acb1-beac-445f-a0ab-5f9280b06116\" (UID: \"9232acb1-beac-445f-a0ab-5f9280b06116\") " Nov 01 23:06:43.711745 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.711622 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9232acb1-beac-445f-a0ab-5f9280b06116-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9232acb1-beac-445f-a0ab-5f9280b06116-plugins-config-volume\") pod \"9232acb1-beac-445f-a0ab-5f9280b06116\" (UID: \"9232acb1-beac-445f-a0ab-5f9280b06116\") " Nov 01 23:06:43.712019 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.711993 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9232acb1-beac-445f-a0ab-5f9280b06116-plugins-config-volume" pod "9232acb1-beac-445f-a0ab-5f9280b06116" (UID: "9232acb1-beac-445f-a0ab-5f9280b06116"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:43.713578 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.713556 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9232acb1-beac-445f-a0ab-5f9280b06116-kube-api-access-v9qtb" pod "9232acb1-beac-445f-a0ab-5f9280b06116" (UID: "9232acb1-beac-445f-a0ab-5f9280b06116"). InnerVolumeSpecName "kube-api-access-v9qtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:43.760271 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.760239 6466 generic.go:356] "Generic (PLEG): container finished" podID="9232acb1-beac-445f-a0ab-5f9280b06116" containerID="24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d" exitCode=0 Nov 01 23:06:43.760423 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.760281 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" event={"ID":"9232acb1-beac-445f-a0ab-5f9280b06116","Type":"ContainerDied","Data":"24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d"} Nov 01 23:06:43.760423 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.760312 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" Nov 01 23:06:43.760423 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.760350 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg" event={"ID":"9232acb1-beac-445f-a0ab-5f9280b06116","Type":"ContainerDied","Data":"aebf2a4b7658c8dcc6c4eb806d62c7501440275e2a823ed0ac5cb06ae4537391"} Nov 01 23:06:43.760423 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.760371 6466 scope.go:122] "RemoveContainer" containerID="24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d" Nov 01 23:06:43.769598 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.769580 6466 scope.go:122] "RemoveContainer" containerID="24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d" Nov 01 23:06:43.769857 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:06:43.769840 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d\": container with ID starting with 24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d not found: ID does not exist" containerID="24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d" Nov 01 23:06:43.769915 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.769864 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d"} err="failed to get container status \"24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d\": rpc error: code = NotFound desc = could not find container \"24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d\": container with ID starting with 24e71b451d7c5bfbf21d0846718f86d55223f767ca11230c424569fddc30fc5d not found: ID does not exist" Nov 01 23:06:43.781731 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.781710 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg"] Nov 01 23:06:43.787943 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.787921 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee0082-745f4xdrkg"] Nov 01 23:06:43.812580 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.812556 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9qtb\" (UniqueName: \"kubernetes.io/projected/9232acb1-beac-445f-a0ab-5f9280b06116-kube-api-access-v9qtb\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:43.812580 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:43.812580 6466 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9232acb1-beac-445f-a0ab-5f9280b06116-plugins-config-volume\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:44.339855 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:44.339819 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9232acb1-beac-445f-a0ab-5f9280b06116" path="/var/lib/kubelet/pods/9232acb1-beac-445f-a0ab-5f9280b06116/volumes" Nov 01 23:06:49.283334 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:06:49.283276 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:06:51.298620 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:51.298228 6466 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-wg8g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:06:51.298620 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:06:51.298304 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-wg8g5" podUID="65c9f274-eb9e-4f06-ae4b-fa127d31c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:07:03.473740 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.473659 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw"] Nov 01 23:07:03.474139 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.474095 6466 cpu_manager.go:414] "containerMap: removing container" podUID="605265bf-8f1a-4910-9d04-d5828c399405" containerName="maas-api" Nov 01 23:07:03.474139 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.474119 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605265bf-8f1a-4910-9d04-d5828c399405" containerName="maas-api" Nov 01 23:07:03.474221 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.474151 6466 cpu_manager.go:414] "containerMap: removing container" podUID="9232acb1-beac-445f-a0ab-5f9280b06116" containerName="payload-pre-processing" Nov 01 23:07:03.474221 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.474156 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9232acb1-beac-445f-a0ab-5f9280b06116" containerName="payload-pre-processing" Nov 01 23:07:03.474293 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.474237 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="605265bf-8f1a-4910-9d04-d5828c399405" containerName="maas-api" Nov 01 23:07:03.474293 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.474250 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="9232acb1-beac-445f-a0ab-5f9280b06116" containerName="payload-pre-processing" Nov 01 23:07:03.478698 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.478677 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:03.483457 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.483415 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:07:03.483696 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.483671 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:07:03.483782 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.483620 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:07:03.483782 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.483577 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:07:03.484244 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.484219 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw"] Nov 01 23:07:03.537033 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.537000 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19969cf-d522-4116-9081-e755269f22d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-l5tkw\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:03.537273 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.537051 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19969cf-d522-4116-9081-e755269f22d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-l5tkw\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:03.537352 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.537270 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7sl\" (UniqueName: \"kubernetes.io/projected/e19969cf-d522-4116-9081-e755269f22d5-kube-api-access-vn7sl\") pod \"maas-api-revoke-sub-452b6d7b-l5tkw\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:03.638583 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.638539 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7sl\" (UniqueName: \"kubernetes.io/projected/e19969cf-d522-4116-9081-e755269f22d5-kube-api-access-vn7sl\") pod \"maas-api-revoke-sub-452b6d7b-l5tkw\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:03.638779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.638602 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19969cf-d522-4116-9081-e755269f22d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-l5tkw\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:03.638779 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.638641 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19969cf-d522-4116-9081-e755269f22d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-l5tkw\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:03.639430 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.639403 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19969cf-d522-4116-9081-e755269f22d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-l5tkw\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:03.641076 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.641056 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19969cf-d522-4116-9081-e755269f22d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-l5tkw\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:03.646944 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.646918 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7sl\" (UniqueName: \"kubernetes.io/projected/e19969cf-d522-4116-9081-e755269f22d5-kube-api-access-vn7sl\") pod \"maas-api-revoke-sub-452b6d7b-l5tkw\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:03.793239 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:03.793135 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:04.077017 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:04.076986 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw"] Nov 01 23:07:04.079029 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:07:04.078992 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19969cf_d522_4116_9081_e755269f22d5.slice/crio-285bb1043a4e917a6a7fea5eaf6ef56f26f15296020a9d3d6fd652cd8a1393e1 WatchSource:0}: Error finding container 285bb1043a4e917a6a7fea5eaf6ef56f26f15296020a9d3d6fd652cd8a1393e1: Status 404 returned error can't find the container with id 285bb1043a4e917a6a7fea5eaf6ef56f26f15296020a9d3d6fd652cd8a1393e1 Nov 01 23:07:04.894316 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:04.894277 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" event={"ID":"e19969cf-d522-4116-9081-e755269f22d5","Type":"ContainerStarted","Data":"dd658257a7aedb6abb64e7d81f246f01d5ea70972b2709f86494cd87812e0b53"} Nov 01 23:07:04.894734 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:04.894338 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" event={"ID":"e19969cf-d522-4116-9081-e755269f22d5","Type":"ContainerStarted","Data":"285bb1043a4e917a6a7fea5eaf6ef56f26f15296020a9d3d6fd652cd8a1393e1"} Nov 01 23:07:04.913887 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:04.913833 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" podStartSLOduration=1.913807926 podStartE2EDuration="1.913807926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:04.91329073 +0000 UTC m=+2628.697729653" watchObservedRunningTime="2026-11-01 23:07:04.913807926 +0000 UTC m=+2628.698246848" Nov 01 23:07:05.908591 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:05.908545 6466 generic.go:356] "Generic (PLEG): container finished" podID="e19969cf-d522-4116-9081-e755269f22d5" containerID="dd658257a7aedb6abb64e7d81f246f01d5ea70972b2709f86494cd87812e0b53" exitCode=0 Nov 01 23:07:05.909170 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:05.908597 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" event={"ID":"e19969cf-d522-4116-9081-e755269f22d5","Type":"ContainerDied","Data":"dd658257a7aedb6abb64e7d81f246f01d5ea70972b2709f86494cd87812e0b53"} Nov 01 23:07:07.301739 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.301662 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:07.430855 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.430820 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e19969cf-d522-4116-9081-e755269f22d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19969cf-d522-4116-9081-e755269f22d5-maas-api-xks-ca\") pod \"e19969cf-d522-4116-9081-e755269f22d5\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " Nov 01 23:07:07.430855 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.430869 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19969cf-d522-4116-9081-e755269f22d5-kube-api-access-vn7sl\" (UniqueName: \"kubernetes.io/projected/e19969cf-d522-4116-9081-e755269f22d5-kube-api-access-vn7sl\") pod \"e19969cf-d522-4116-9081-e755269f22d5\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " Nov 01 23:07:07.431147 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.430965 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e19969cf-d522-4116-9081-e755269f22d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19969cf-d522-4116-9081-e755269f22d5-maas-api-openshift-ca\") pod \"e19969cf-d522-4116-9081-e755269f22d5\" (UID: \"e19969cf-d522-4116-9081-e755269f22d5\") " Nov 01 23:07:07.431417 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.431380 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19969cf-d522-4116-9081-e755269f22d5-maas-api-openshift-ca" pod "e19969cf-d522-4116-9081-e755269f22d5" (UID: "e19969cf-d522-4116-9081-e755269f22d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:07.431554 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.431534 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19969cf-d522-4116-9081-e755269f22d5-maas-api-openshift-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:07.433001 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.432982 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19969cf-d522-4116-9081-e755269f22d5-maas-api-xks-ca" pod "e19969cf-d522-4116-9081-e755269f22d5" (UID: "e19969cf-d522-4116-9081-e755269f22d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:07.433078 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.433053 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19969cf-d522-4116-9081-e755269f22d5-kube-api-access-vn7sl" pod "e19969cf-d522-4116-9081-e755269f22d5" (UID: "e19969cf-d522-4116-9081-e755269f22d5"). InnerVolumeSpecName "kube-api-access-vn7sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:07.532844 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.532752 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19969cf-d522-4116-9081-e755269f22d5-maas-api-xks-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:07.532844 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.532785 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn7sl\" (UniqueName: \"kubernetes.io/projected/e19969cf-d522-4116-9081-e755269f22d5-kube-api-access-vn7sl\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:07.921268 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.921237 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" Nov 01 23:07:07.921448 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.921239 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw" event={"ID":"e19969cf-d522-4116-9081-e755269f22d5","Type":"ContainerDied","Data":"285bb1043a4e917a6a7fea5eaf6ef56f26f15296020a9d3d6fd652cd8a1393e1"} Nov 01 23:07:07.921448 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:07.921357 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="285bb1043a4e917a6a7fea5eaf6ef56f26f15296020a9d3d6fd652cd8a1393e1" Nov 01 23:07:19.320406 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:07:19.320369 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:07:19.547650 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.547614 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7"] Nov 01 23:07:19.548100 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.548085 6466 cpu_manager.go:414] "containerMap: removing container" podUID="e19969cf-d522-4116-9081-e755269f22d5" containerName="revoke-keys" Nov 01 23:07:19.548181 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.548118 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19969cf-d522-4116-9081-e755269f22d5" containerName="revoke-keys" Nov 01 23:07:19.548230 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.548219 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19969cf-d522-4116-9081-e755269f22d5" containerName="revoke-keys" Nov 01 23:07:19.551510 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.551494 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:19.558181 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.558102 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:07:19.558181 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.558135 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:07:19.558181 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.558158 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:07:19.558463 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.558259 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:07:19.560641 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.560614 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7"] Nov 01 23:07:19.691901 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.691866 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9msg\" (UniqueName: \"kubernetes.io/projected/f43085e1-d107-4fce-a334-dcd149a4465c-kube-api-access-p9msg\") pod \"maas-api-revoke-sub-3e31a506-wjbp7\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:19.692084 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.692003 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-wjbp7\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:19.692171 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.692151 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-wjbp7\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:19.796415 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.796362 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-wjbp7\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:19.796626 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.796484 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9msg\" (UniqueName: \"kubernetes.io/projected/f43085e1-d107-4fce-a334-dcd149a4465c-kube-api-access-p9msg\") pod \"maas-api-revoke-sub-3e31a506-wjbp7\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:19.796702 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.796644 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-wjbp7\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:19.797608 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.797575 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-wjbp7\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:19.801979 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.801948 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-wjbp7\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:19.807922 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.807889 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9msg\" (UniqueName: \"kubernetes.io/projected/f43085e1-d107-4fce-a334-dcd149a4465c-kube-api-access-p9msg\") pod \"maas-api-revoke-sub-3e31a506-wjbp7\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:19.867246 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:19.867208 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:20.161333 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:20.161302 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7"] Nov 01 23:07:20.166982 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:07:20.166018 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43085e1_d107_4fce_a334_dcd149a4465c.slice/crio-b4d7eaf04d78f23f84d3d289e8ad61ef13e5246204617abec3103927b49deaec WatchSource:0}: Error finding container b4d7eaf04d78f23f84d3d289e8ad61ef13e5246204617abec3103927b49deaec: Status 404 returned error can't find the container with id b4d7eaf04d78f23f84d3d289e8ad61ef13e5246204617abec3103927b49deaec Nov 01 23:07:21.020020 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:21.019916 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" event={"ID":"f43085e1-d107-4fce-a334-dcd149a4465c","Type":"ContainerStarted","Data":"ffae4c9aec2c00c5dca5651db54dc45ee42554a45881fd245250d3aa515cab3e"} Nov 01 23:07:21.021339 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:21.020072 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" event={"ID":"f43085e1-d107-4fce-a334-dcd149a4465c","Type":"ContainerStarted","Data":"b4d7eaf04d78f23f84d3d289e8ad61ef13e5246204617abec3103927b49deaec"} Nov 01 23:07:21.080747 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:21.067319 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" podStartSLOduration=2.067297884 podStartE2EDuration="2.067297884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:21.053486521 +0000 UTC m=+2644.837925445" watchObservedRunningTime="2026-11-01 23:07:21.067297884 +0000 UTC m=+2644.851736808" Nov 01 23:07:22.026265 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:22.026230 6466 generic.go:356] "Generic (PLEG): container finished" podID="f43085e1-d107-4fce-a334-dcd149a4465c" containerID="ffae4c9aec2c00c5dca5651db54dc45ee42554a45881fd245250d3aa515cab3e" exitCode=0 Nov 01 23:07:22.026713 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:22.026322 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" event={"ID":"f43085e1-d107-4fce-a334-dcd149a4465c","Type":"ContainerDied","Data":"ffae4c9aec2c00c5dca5651db54dc45ee42554a45881fd245250d3aa515cab3e"} Nov 01 23:07:23.167980 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:23.167946 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:23.236986 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:23.236957 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-xks-ca\") pod \"f43085e1-d107-4fce-a334-dcd149a4465c\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " Nov 01 23:07:23.237219 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:23.237049 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-openshift-ca\") pod \"f43085e1-d107-4fce-a334-dcd149a4465c\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " Nov 01 23:07:23.237219 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:23.237148 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43085e1-d107-4fce-a334-dcd149a4465c-kube-api-access-p9msg\" (UniqueName: \"kubernetes.io/projected/f43085e1-d107-4fce-a334-dcd149a4465c-kube-api-access-p9msg\") pod \"f43085e1-d107-4fce-a334-dcd149a4465c\" (UID: \"f43085e1-d107-4fce-a334-dcd149a4465c\") " Nov 01 23:07:23.237687 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:23.237657 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-openshift-ca" pod "f43085e1-d107-4fce-a334-dcd149a4465c" (UID: "f43085e1-d107-4fce-a334-dcd149a4465c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:23.239286 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:23.239262 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-xks-ca" pod "f43085e1-d107-4fce-a334-dcd149a4465c" (UID: "f43085e1-d107-4fce-a334-dcd149a4465c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:23.239742 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:23.239721 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43085e1-d107-4fce-a334-dcd149a4465c-kube-api-access-p9msg" pod "f43085e1-d107-4fce-a334-dcd149a4465c" (UID: "f43085e1-d107-4fce-a334-dcd149a4465c"). InnerVolumeSpecName "kube-api-access-p9msg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:23.338218 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:23.338138 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9msg\" (UniqueName: \"kubernetes.io/projected/f43085e1-d107-4fce-a334-dcd149a4465c-kube-api-access-p9msg\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:23.338218 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:23.338167 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-xks-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:23.338218 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:23.338176 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43085e1-d107-4fce-a334-dcd149a4465c-maas-api-openshift-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:24.038705 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:24.038668 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" event={"ID":"f43085e1-d107-4fce-a334-dcd149a4465c","Type":"ContainerDied","Data":"b4d7eaf04d78f23f84d3d289e8ad61ef13e5246204617abec3103927b49deaec"} Nov 01 23:07:24.038892 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:24.038715 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7" Nov 01 23:07:24.038892 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:24.038719 6466 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4d7eaf04d78f23f84d3d289e8ad61ef13e5246204617abec3103927b49deaec" Nov 01 23:07:49.280064 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:07:49.280027 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:07:51.729289 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.729256 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c"] Nov 01 23:07:51.729763 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.729702 6466 cpu_manager.go:414] "containerMap: removing container" podUID="f43085e1-d107-4fce-a334-dcd149a4465c" containerName="revoke-keys" Nov 01 23:07:51.729763 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.729713 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43085e1-d107-4fce-a334-dcd149a4465c" containerName="revoke-keys" Nov 01 23:07:51.729845 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.729781 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43085e1-d107-4fce-a334-dcd149a4465c" containerName="revoke-keys" Nov 01 23:07:51.733060 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.733038 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.741568 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.741529 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c"] Nov 01 23:07:51.784321 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.784286 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 23:07:51.784321 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.784286 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 23:07:51.784534 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.784346 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7x7m\"" Nov 01 23:07:51.822038 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.822012 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:07:51.822233 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.822071 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:07:51.873590 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.873552 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/338da0a4-f784-4869-845d-078a29a3efe2-openshift-service-ca\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.873774 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.873697 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/338da0a4-f784-4869-845d-078a29a3efe2-maas-api-tls\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.873774 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.873745 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkppf\" (UniqueName: \"kubernetes.io/projected/338da0a4-f784-4869-845d-078a29a3efe2-kube-api-access-wkppf\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.873774 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.873771 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/338da0a4-f784-4869-845d-078a29a3efe2-metrics-certs\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.974702 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.974668 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/338da0a4-f784-4869-845d-078a29a3efe2-maas-api-tls\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.974889 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.974727 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkppf\" (UniqueName: \"kubernetes.io/projected/338da0a4-f784-4869-845d-078a29a3efe2-kube-api-access-wkppf\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.974889 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.974764 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/338da0a4-f784-4869-845d-078a29a3efe2-metrics-certs\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.974889 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.974818 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/338da0a4-f784-4869-845d-078a29a3efe2-openshift-service-ca\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.975617 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.975593 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/338da0a4-f784-4869-845d-078a29a3efe2-openshift-service-ca\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.977412 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.977388 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/338da0a4-f784-4869-845d-078a29a3efe2-maas-api-tls\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.977522 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.977417 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/338da0a4-f784-4869-845d-078a29a3efe2-metrics-certs\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:51.983691 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:51.983638 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkppf\" (UniqueName: \"kubernetes.io/projected/338da0a4-f784-4869-845d-078a29a3efe2-kube-api-access-wkppf\") pod \"maas-api-5fdf88964b-6gl2c\" (UID: \"338da0a4-f784-4869-845d-078a29a3efe2\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:52.048787 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:52.048751 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:52.328047 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:52.328020 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c"] Nov 01 23:07:52.330547 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:07:52.330514 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338da0a4_f784_4869_845d_078a29a3efe2.slice/crio-5d1e93998da640af1848ef4beeabaf1954bba3d931b2b63cc6c6afb650e09cb4 WatchSource:0}: Error finding container 5d1e93998da640af1848ef4beeabaf1954bba3d931b2b63cc6c6afb650e09cb4: Status 404 returned error can't find the container with id 5d1e93998da640af1848ef4beeabaf1954bba3d931b2b63cc6c6afb650e09cb4 Nov 01 23:07:52.331838 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:52.331819 6466 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:07:53.211184 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:53.211142 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" event={"ID":"338da0a4-f784-4869-845d-078a29a3efe2","Type":"ContainerStarted","Data":"bc00215f39f886eef16fca3a755f4b405f6afa426681f6530cf7a52248cef996"} Nov 01 23:07:53.211602 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:53.211204 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" event={"ID":"338da0a4-f784-4869-845d-078a29a3efe2","Type":"ContainerStarted","Data":"5d1e93998da640af1848ef4beeabaf1954bba3d931b2b63cc6c6afb650e09cb4"} Nov 01 23:07:53.211602 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:53.211376 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:07:53.234528 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:53.234471 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" podStartSLOduration=1.800676817 podStartE2EDuration="2.234453437s" totalImagesPullingTime="433.77662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:53.232848129 +0000 UTC m=+2677.017287046" watchObservedRunningTime="2026-11-01 23:07:53.234453437 +0000 UTC m=+2677.018892361" Nov 01 23:07:59.222214 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:07:59.222184 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-6gl2c" Nov 01 23:08:02.430585 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.430551 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc"] Nov 01 23:08:02.435532 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.435502 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:02.445562 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.445534 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc"] Nov 01 23:08:02.447310 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.447279 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-75893881-serving-cert\"" Nov 01 23:08:02.488622 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.488585 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5b203c-902e-4cc6-b455-60d30341d7f9-openshift-service-ca\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:02.488838 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.488646 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-maas-api-tls\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:02.488838 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.488761 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-metrics-certs\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:02.488838 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.488830 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blbhw\" (UniqueName: \"kubernetes.io/projected/ab5b203c-902e-4cc6-b455-60d30341d7f9-kube-api-access-blbhw\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:02.589970 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.589933 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-maas-api-tls\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:02.590210 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.589994 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-metrics-certs\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:02.590210 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.590025 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blbhw\" (UniqueName: \"kubernetes.io/projected/ab5b203c-902e-4cc6-b455-60d30341d7f9-kube-api-access-blbhw\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:02.590210 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.590098 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5b203c-902e-4cc6-b455-60d30341d7f9-openshift-service-ca\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:02.590210 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:08:02.590133 6466 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-75893881-serving-cert: secret "maas-api-e2e-ait-75893881-serving-cert" not found Nov 01 23:08:02.590210 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:08:02.590213 6466 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-maas-api-tls podName:ab5b203c-902e-4cc6-b455-60d30341d7f9 nodeName:}" failed. No retries permitted until 2026-11-01 23:08:03.090192833 +0000 UTC m=+2686.874631737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-maas-api-tls") pod "maas-api-e2e-ait-75893881-6678f9549d-fjvxc" (UID: "ab5b203c-902e-4cc6-b455-60d30341d7f9") : secret "maas-api-e2e-ait-75893881-serving-cert" not found Nov 01 23:08:02.590830 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.590812 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5b203c-902e-4cc6-b455-60d30341d7f9-openshift-service-ca\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:02.592529 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.592510 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-metrics-certs\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:02.606274 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:02.606246 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blbhw\" (UniqueName: \"kubernetes.io/projected/ab5b203c-902e-4cc6-b455-60d30341d7f9-kube-api-access-blbhw\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:03.096727 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:03.096687 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-maas-api-tls\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:03.099210 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:03.099184 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-maas-api-tls\") pod \"maas-api-e2e-ait-75893881-6678f9549d-fjvxc\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:03.353638 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:03.353546 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:03.631188 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:03.631160 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc"] Nov 01 23:08:03.633330 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:08:03.633300 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5b203c_902e_4cc6_b455_60d30341d7f9.slice/crio-57aa8ff5e8208ebfc17aa08d61cfbefbcba89b3cef9c87ccc93c005e96587946 WatchSource:0}: Error finding container 57aa8ff5e8208ebfc17aa08d61cfbefbcba89b3cef9c87ccc93c005e96587946: Status 404 returned error can't find the container with id 57aa8ff5e8208ebfc17aa08d61cfbefbcba89b3cef9c87ccc93c005e96587946 Nov 01 23:08:04.274461 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:04.274423 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" event={"ID":"ab5b203c-902e-4cc6-b455-60d30341d7f9","Type":"ContainerStarted","Data":"1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f"} Nov 01 23:08:04.274666 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:04.274487 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" event={"ID":"ab5b203c-902e-4cc6-b455-60d30341d7f9","Type":"ContainerStarted","Data":"57aa8ff5e8208ebfc17aa08d61cfbefbcba89b3cef9c87ccc93c005e96587946"} Nov 01 23:08:04.274666 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:04.274603 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:04.296162 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:04.296088 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" podStartSLOduration=1.871387141 podStartE2EDuration="2.296072455s" totalImagesPullingTime="424.685314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:04.295268348 +0000 UTC m=+2688.079707270" watchObservedRunningTime="2026-11-01 23:08:04.296072455 +0000 UTC m=+2688.080511375" Nov 01 23:08:10.286031 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:10.286001 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:19.282523 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:08:19.282484 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:08:27.733189 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:27.733065 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc"] Nov 01 23:08:27.733860 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:27.733356 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" podUID="ab5b203c-902e-4cc6-b455-60d30341d7f9" containerName="maas-api" containerID="cri-o://1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f" gracePeriod=30 Nov 01 23:08:28.016478 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.016451 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:28.102867 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.102832 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-metrics-certs\") pod \"ab5b203c-902e-4cc6-b455-60d30341d7f9\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " Nov 01 23:08:28.103064 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.102884 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5b203c-902e-4cc6-b455-60d30341d7f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5b203c-902e-4cc6-b455-60d30341d7f9-openshift-service-ca\") pod \"ab5b203c-902e-4cc6-b455-60d30341d7f9\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " Nov 01 23:08:28.103064 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.103023 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5b203c-902e-4cc6-b455-60d30341d7f9-kube-api-access-blbhw\" (UniqueName: \"kubernetes.io/projected/ab5b203c-902e-4cc6-b455-60d30341d7f9-kube-api-access-blbhw\") pod \"ab5b203c-902e-4cc6-b455-60d30341d7f9\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " Nov 01 23:08:28.103233 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.103173 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-maas-api-tls\") pod \"ab5b203c-902e-4cc6-b455-60d30341d7f9\" (UID: \"ab5b203c-902e-4cc6-b455-60d30341d7f9\") " Nov 01 23:08:28.103301 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.103281 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5b203c-902e-4cc6-b455-60d30341d7f9-openshift-service-ca" pod "ab5b203c-902e-4cc6-b455-60d30341d7f9" (UID: "ab5b203c-902e-4cc6-b455-60d30341d7f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:28.103565 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.103539 6466 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5b203c-902e-4cc6-b455-60d30341d7f9-openshift-service-ca\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:28.105330 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.105304 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-maas-api-tls" pod "ab5b203c-902e-4cc6-b455-60d30341d7f9" (UID: "ab5b203c-902e-4cc6-b455-60d30341d7f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:28.105524 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.105500 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5b203c-902e-4cc6-b455-60d30341d7f9-kube-api-access-blbhw" pod "ab5b203c-902e-4cc6-b455-60d30341d7f9" (UID: "ab5b203c-902e-4cc6-b455-60d30341d7f9"). InnerVolumeSpecName "kube-api-access-blbhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:28.105795 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.105764 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-metrics-certs" pod "ab5b203c-902e-4cc6-b455-60d30341d7f9" (UID: "ab5b203c-902e-4cc6-b455-60d30341d7f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:28.204417 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.204330 6466 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-metrics-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:28.204417 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.204371 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blbhw\" (UniqueName: \"kubernetes.io/projected/ab5b203c-902e-4cc6-b455-60d30341d7f9-kube-api-access-blbhw\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:28.204417 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.204389 6466 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5b203c-902e-4cc6-b455-60d30341d7f9-maas-api-tls\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:28.416311 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.416277 6466 generic.go:356] "Generic (PLEG): container finished" podID="ab5b203c-902e-4cc6-b455-60d30341d7f9" containerID="1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f" exitCode=0 Nov 01 23:08:28.416492 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.416329 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" Nov 01 23:08:28.416492 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.416364 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" event={"ID":"ab5b203c-902e-4cc6-b455-60d30341d7f9","Type":"ContainerDied","Data":"1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f"} Nov 01 23:08:28.416492 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.416421 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc" event={"ID":"ab5b203c-902e-4cc6-b455-60d30341d7f9","Type":"ContainerDied","Data":"57aa8ff5e8208ebfc17aa08d61cfbefbcba89b3cef9c87ccc93c005e96587946"} Nov 01 23:08:28.416492 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.416451 6466 scope.go:122] "RemoveContainer" containerID="1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f" Nov 01 23:08:28.427861 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.427839 6466 scope.go:122] "RemoveContainer" containerID="1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f" Nov 01 23:08:28.428228 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:08:28.428197 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f\": container with ID starting with 1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f not found: ID does not exist" containerID="1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f" Nov 01 23:08:28.428289 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.428241 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f"} err="failed to get container status \"1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f\": rpc error: code = NotFound desc = could not find container \"1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f\": container with ID starting with 1f50bf3d523cf9fa5c5c18f39e3c2c1eb3759d697de343a715b49700c330840f not found: ID does not exist" Nov 01 23:08:28.433920 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.433892 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc"] Nov 01 23:08:28.439352 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:28.439327 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75893881-6678f9549d-fjvxc"] Nov 01 23:08:30.338996 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:30.338963 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5b203c-902e-4cc6-b455-60d30341d7f9" path="/var/lib/kubelet/pods/ab5b203c-902e-4cc6-b455-60d30341d7f9/volumes" Nov 01 23:08:34.673660 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.673618 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74"] Nov 01 23:08:34.674533 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.674495 6466 cpu_manager.go:414] "containerMap: removing container" podUID="ab5b203c-902e-4cc6-b455-60d30341d7f9" containerName="maas-api" Nov 01 23:08:34.674533 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.674526 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5b203c-902e-4cc6-b455-60d30341d7f9" containerName="maas-api" Nov 01 23:08:34.675409 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.675383 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5b203c-902e-4cc6-b455-60d30341d7f9" containerName="maas-api" Nov 01 23:08:34.679298 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.679284 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.691368 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.691343 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-4b45d6-openshift-default-dockercfg-hvdk2\"" Nov 01 23:08:34.692200 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.692179 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74"] Nov 01 23:08:34.775849 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.775806 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-credential-socket\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.776056 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.775878 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5vp2\" (UniqueName: \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-kube-api-access-r5vp2\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.776056 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.775923 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-certs\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.776056 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.775955 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istiod-ca-cert\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.776056 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.775993 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-token\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.776338 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.776063 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-envoy\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.776338 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.776141 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-data\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.776338 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.776204 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-socket\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.776338 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.776237 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-podinfo\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.876923 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.876887 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-socket\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.876923 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.876931 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-podinfo\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.877246 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.876966 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-credential-socket\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.877246 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.876991 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5vp2\" (UniqueName: \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-kube-api-access-r5vp2\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.877246 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.877015 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-certs\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.877246 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.877038 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istiod-ca-cert\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.877246 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.877076 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-token\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.877246 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.877167 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-envoy\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.877246 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.877223 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-data\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.877609 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.877364 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-socket\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.877609 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.877441 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-credential-socket\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.877725 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.877620 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-certs\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.877725 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.877690 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-data\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.878078 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.878045 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istiod-ca-cert\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.879611 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.879582 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-podinfo\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.879703 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.879585 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-envoy\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.890978 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.890952 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-token\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.891202 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.891181 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5vp2\" (UniqueName: \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-kube-api-access-r5vp2\") pod \"e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:34.996681 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:34.996592 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:35.276687 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:35.276658 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74"] Nov 01 23:08:35.278602 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:08:35.278572 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48108ced_26bd_4bfd_832e_b14e3f36c1f8.slice/crio-02dac27fc2ec130f04a27871c39fb618b86338280aadb5cf338a843ecefd129d WatchSource:0}: Error finding container 02dac27fc2ec130f04a27871c39fb618b86338280aadb5cf338a843ecefd129d: Status 404 returned error can't find the container with id 02dac27fc2ec130f04a27871c39fb618b86338280aadb5cf338a843ecefd129d Nov 01 23:08:35.280799 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:35.280766 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:08:35.280914 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:35.280847 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:08:35.280914 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:35.280893 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:08:35.455259 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:35.455217 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" event={"ID":"48108ced-26bd-4bfd-832e-b14e3f36c1f8","Type":"ContainerStarted","Data":"0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093"} Nov 01 23:08:35.455444 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:35.455283 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" event={"ID":"48108ced-26bd-4bfd-832e-b14e3f36c1f8","Type":"ContainerStarted","Data":"02dac27fc2ec130f04a27871c39fb618b86338280aadb5cf338a843ecefd129d"} Nov 01 23:08:35.479854 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:35.479793 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" podStartSLOduration=1.479772963 podStartE2EDuration="1.479772963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:35.479273631 +0000 UTC m=+2719.263712554" watchObservedRunningTime="2026-11-01 23:08:35.479772963 +0000 UTC m=+2719.264211886" Nov 01 23:08:35.997518 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:35.997483 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:37.003046 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:37.003015 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:37.464684 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:37.464656 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:37.465635 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:37.465615 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:49.287297 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:08:49.287260 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:08:51.573825 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:51.573792 6466 generic.go:356] "Generic (PLEG): container finished" podID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerID="0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093" exitCode=0 Nov 01 23:08:51.574282 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:51.573847 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" event={"ID":"48108ced-26bd-4bfd-832e-b14e3f36c1f8","Type":"ContainerDied","Data":"0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093"} Nov 01 23:08:51.574282 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:51.574233 6466 scope.go:122] "RemoveContainer" containerID="0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093" Nov 01 23:08:51.575537 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:51.575508 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:08:51.575616 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:51.575579 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:08:51.575616 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:51.575608 6466 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"} Nov 01 23:08:52.465043 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:52.464997 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:52.580444 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:52.580405 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" event={"ID":"48108ced-26bd-4bfd-832e-b14e3f36c1f8","Type":"ContainerStarted","Data":"7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db"} Nov 01 23:08:52.996825 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:52.996792 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:52.998079 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:52.998052 6466 patch_prober.go:29] interesting pod/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 23:08:52.998261 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:52.998139 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 23:08:53.997194 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:53.997151 6466 patch_prober.go:29] interesting pod/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 23:08:53.997632 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:53.997224 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 23:08:54.997922 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:54.997878 6466 patch_prober.go:29] interesting pod/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 23:08:54.998383 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:54.997940 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 23:08:55.997738 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:55.997703 6466 patch_prober.go:29] interesting pod/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 23:08:55.997932 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:55.997756 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 23:08:56.997561 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:56.997523 6466 patch_prober.go:29] interesting pod/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 01 23:08:56.997977 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:56.997580 6466 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 01 23:08:58.001560 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:58.001524 6466 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:58.002024 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:58.001820 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:08:58.002640 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:08:58.002623 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:09:19.281926 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:09:19.281883 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:09:49.280916 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:09:49.280872 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:10:19.281247 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:10:19.281205 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:10:38.738924 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:38.738884 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74"] Nov 01 23:10:38.739524 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:38.739093 6466 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" containerID="cri-o://7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db" gracePeriod=30 Nov 01 23:10:43.009546 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:43.009504 6466 patch_prober.go:29] interesting pod/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:10:43.010229 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:43.010194 6466 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:10:44.004772 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.004697 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:10:44.095944 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.095912 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-credential-socket\") pod \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " Nov 01 23:10:44.096465 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.095956 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-token\" (UniqueName: \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-token\") pod \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " Nov 01 23:10:44.096465 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.095993 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istiod-ca-cert\") pod \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " Nov 01 23:10:44.096465 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096020 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-data\") pod \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " Nov 01 23:10:44.096465 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096046 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-certs\") pod \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " Nov 01 23:10:44.096465 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096081 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-socket\") pod \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " Nov 01 23:10:44.096465 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096240 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-credential-socket" pod "48108ced-26bd-4bfd-832e-b14e3f36c1f8" (UID: "48108ced-26bd-4bfd-832e-b14e3f36c1f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:44.096465 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096245 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-kube-api-access-r5vp2\" (UniqueName: \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-kube-api-access-r5vp2\") pod \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " Nov 01 23:10:44.096465 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096347 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-envoy\") pod \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " Nov 01 23:10:44.096465 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096389 6466 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-podinfo\") pod \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\" (UID: \"48108ced-26bd-4bfd-832e-b14e3f36c1f8\") " Nov 01 23:10:44.096853 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096385 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-certs" pod "48108ced-26bd-4bfd-832e-b14e3f36c1f8" (UID: "48108ced-26bd-4bfd-832e-b14e3f36c1f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:44.096853 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096394 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-data" pod "48108ced-26bd-4bfd-832e-b14e3f36c1f8" (UID: "48108ced-26bd-4bfd-832e-b14e3f36c1f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:44.096853 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096507 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-socket" pod "48108ced-26bd-4bfd-832e-b14e3f36c1f8" (UID: "48108ced-26bd-4bfd-832e-b14e3f36c1f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:44.096853 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096531 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istiod-ca-cert" pod "48108ced-26bd-4bfd-832e-b14e3f36c1f8" (UID: "48108ced-26bd-4bfd-832e-b14e3f36c1f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:44.096853 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096834 6466 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istiod-ca-cert\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.097040 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096858 6466 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-data\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.097040 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096873 6466 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-certs\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.097040 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096887 6466 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-workload-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.097040 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.096904 6466 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-credential-socket\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.098501 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.098474 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-envoy" pod "48108ced-26bd-4bfd-832e-b14e3f36c1f8" (UID: "48108ced-26bd-4bfd-832e-b14e3f36c1f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:44.098620 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.098506 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-podinfo" pod "48108ced-26bd-4bfd-832e-b14e3f36c1f8" (UID: "48108ced-26bd-4bfd-832e-b14e3f36c1f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:44.098818 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.098792 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-token" pod "48108ced-26bd-4bfd-832e-b14e3f36c1f8" (UID: "48108ced-26bd-4bfd-832e-b14e3f36c1f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:44.098914 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.098821 6466 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-kube-api-access-r5vp2" pod "48108ced-26bd-4bfd-832e-b14e3f36c1f8" (UID: "48108ced-26bd-4bfd-832e-b14e3f36c1f8"). InnerVolumeSpecName "kube-api-access-r5vp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:44.190879 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.190845 6466 generic.go:356] "Generic (PLEG): container finished" podID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerID="7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db" exitCode=0 Nov 01 23:10:44.191053 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.190902 6466 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" Nov 01 23:10:44.191053 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.190928 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" event={"ID":"48108ced-26bd-4bfd-832e-b14e3f36c1f8","Type":"ContainerDied","Data":"7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db"} Nov 01 23:10:44.191053 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.190987 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74" event={"ID":"48108ced-26bd-4bfd-832e-b14e3f36c1f8","Type":"ContainerDied","Data":"02dac27fc2ec130f04a27871c39fb618b86338280aadb5cf338a843ecefd129d"} Nov 01 23:10:44.191053 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.191015 6466 scope.go:122] "RemoveContainer" containerID="7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db" Nov 01 23:10:44.197610 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.197583 6466 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5vp2\" (UniqueName: \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-kube-api-access-r5vp2\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.197737 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.197613 6466 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-envoy\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.197737 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.197628 6466 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-podinfo\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.197737 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.197640 6466 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48108ced-26bd-4bfd-832e-b14e3f36c1f8-istio-token\") on node \"ip-10-0-134-88.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.200894 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.200877 6466 scope.go:122] "RemoveContainer" containerID="0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093" Nov 01 23:10:44.209037 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.209018 6466 scope.go:122] "RemoveContainer" containerID="7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db" Nov 01 23:10:44.209511 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:10:44.209479 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db\": container with ID starting with 7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db not found: ID does not exist" containerID="7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db" Nov 01 23:10:44.209620 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.209523 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db"} err="failed to get container status \"7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db\": rpc error: code = NotFound desc = could not find container \"7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db\": container with ID starting with 7009beabba6251ebf3e13d0043ae7131c0f46b7279663c538e7d8e99ba7886db not found: ID does not exist" Nov 01 23:10:44.209620 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.209546 6466 scope.go:122] "RemoveContainer" containerID="0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093" Nov 01 23:10:44.209854 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:10:44.209827 6466 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093\": container with ID starting with 0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093 not found: ID does not exist" containerID="0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093" Nov 01 23:10:44.209920 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.209863 6466 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093"} err="failed to get container status \"0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093\": rpc error: code = NotFound desc = could not find container \"0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093\": container with ID starting with 0da612a878068098e3adc4f319cc139dfc3a1747f354069fa9213b7f52178093 not found: ID does not exist" Nov 01 23:10:44.227613 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.227582 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74"] Nov 01 23:10:44.241973 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.241922 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-4b45d6-openshift-default-65674f5589-mbn74"] Nov 01 23:10:44.340812 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:10:44.340723 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" path="/var/lib/kubelet/pods/48108ced-26bd-4bfd-832e-b14e3f36c1f8/volumes" Nov 01 23:10:49.279430 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:10:49.279388 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:11:19.280354 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:11:19.280316 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:11:38.548229 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:38.548139 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ngj6m_51eead1c-88d4-4e16-b360-6f370b3041b8/manager/0.log" Nov 01 23:11:38.716866 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:38.716828 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-x5z49_56efc165-48d0-46f6-ad1a-f6cf91fc3833/manager/0.log" Nov 01 23:11:39.158823 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:39.158777 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b57d74f86-55cvs_c30bab32-ebb2-4bc9-93b4-9cd67b593e4f/manager/0.log" Nov 01 23:11:39.322046 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:39.322011 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-dcdfx_cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc/server/0.log" Nov 01 23:11:39.504020 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:39.503940 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-rs5t6_b2094375-d4ac-4cf1-a8da-bb4e90ba110c/manager/0.log" Nov 01 23:11:40.599340 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:40.599301 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fdf88964b-6gl2c_338da0a4-f784-4869-845d-078a29a3efe2/maas-api/0.log" Nov 01 23:11:41.121232 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:41.121195 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-wjbp7_f43085e1-d107-4fce-a334-dcd149a4465c/revoke-keys/0.log" Nov 01 23:11:41.285634 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:41.285595 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-l5tkw_e19969cf-d522-4116-9081-e755269f22d5/revoke-keys/0.log" Nov 01 23:11:43.878119 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:43.878074 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fqs5h_1f4e4413-8e24-4a65-937d-cc4a9bc0bc25/kuadrant-console-plugin/0.log" Nov 01 23:11:45.552962 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:45.552920 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vpkhv_6312a0f9-4dea-44d8-94a5-35f1ee409f10/discovery/0.log" Nov 01 23:11:45.717781 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:45.717748 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d9489c85-8f7t9_3cea7bcd-f62d-4528-b266-4d3a022c7de3/kube-auth-proxy/0.log" Nov 01 23:11:46.059409 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:46.059378 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-44l6s_5a202d72-2f71-4a71-aba6-df821c076507/istio-proxy/0.log" Nov 01 23:11:47.427026 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:47.426992 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-wfrzw_06103ca9-e37d-4286-960e-cd1c06da76af/istio-proxy/1.log" Nov 01 23:11:47.598187 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:47.598084 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-wg8g5_65c9f274-eb9e-4f06-ae4b-fa127d31c463/istio-proxy/0.log" Nov 01 23:11:47.919400 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:47.919372 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rvxmv_96c96ea8-9914-43f7-a349-2a8322a7839c/payload-pre-processing/0.log" Nov 01 23:11:49.185256 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:49.185226 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-wfrzw_06103ca9-e37d-4286-960e-cd1c06da76af/istio-proxy/0.log" Nov 01 23:11:49.281049 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:11:49.281014 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:11:49.687053 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:49.687020 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5758p_20f8556e-bad5-456f-ad1a-14ef9a2ea949/main/0.log" Nov 01 23:11:50.001370 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:50.001292 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-qkfnz_58de7f57-d10e-42ff-ab3e-3952b601bfce/main/0.log" Nov 01 23:11:50.154344 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:50.154313 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-lxpj5_ce3f4a45-aa4a-465f-a436-9c0f4a542380/main/0.log" Nov 01 23:11:50.316484 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:50.316394 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h_ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e/main/0.log" Nov 01 23:11:50.489160 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:50.489127 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-7klkb_66937625-7220-44c9-b11a-dbc627d0219c/main/0.log" Nov 01 23:11:50.664729 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:11:50.664695 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh_7006c77f-be7e-43be-b47d-7aaef13e5cc6/main/0.log" Nov 01 23:12:00.407138 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:00.407093 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ngj6m_51eead1c-88d4-4e16-b360-6f370b3041b8/manager/0.log" Nov 01 23:12:00.595048 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:00.595015 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-x5z49_56efc165-48d0-46f6-ad1a-f6cf91fc3833/manager/0.log" Nov 01 23:12:00.981384 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:00.981343 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b57d74f86-55cvs_c30bab32-ebb2-4bc9-93b4-9cd67b593e4f/manager/0.log" Nov 01 23:12:01.170648 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:01.170623 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-dcdfx_cd830101-1bd1-4ce9-aae2-eeac9ca7d5fc/server/0.log" Nov 01 23:12:01.352080 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:01.352001 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-rs5t6_b2094375-d4ac-4cf1-a8da-bb4e90ba110c/manager/0.log" Nov 01 23:12:02.032912 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:02.032883 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fdf88964b-6gl2c_338da0a4-f784-4869-845d-078a29a3efe2/maas-api/0.log" Nov 01 23:12:02.574403 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:02.574372 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-wjbp7_f43085e1-d107-4fce-a334-dcd149a4465c/revoke-keys/0.log" Nov 01 23:12:02.744046 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:02.744013 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-l5tkw_e19969cf-d522-4116-9081-e755269f22d5/revoke-keys/0.log" Nov 01 23:12:05.066331 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:05.066301 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fqs5h_1f4e4413-8e24-4a65-937d-cc4a9bc0bc25/kuadrant-console-plugin/0.log" Nov 01 23:12:06.252730 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:06.252702 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vpkhv_6312a0f9-4dea-44d8-94a5-35f1ee409f10/discovery/0.log" Nov 01 23:12:06.424650 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:06.424620 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d9489c85-8f7t9_3cea7bcd-f62d-4528-b266-4d3a022c7de3/kube-auth-proxy/0.log" Nov 01 23:12:06.760120 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:06.760075 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-44l6s_5a202d72-2f71-4a71-aba6-df821c076507/istio-proxy/0.log" Nov 01 23:12:07.020426 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:07.020339 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw"] Nov 01 23:12:07.022613 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:07.022585 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-l5tkw"] Nov 01 23:12:08.115393 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:08.115365 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-wfrzw_06103ca9-e37d-4286-960e-cd1c06da76af/istio-proxy/1.log" Nov 01 23:12:08.281535 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:08.281493 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-wg8g5_65c9f274-eb9e-4f06-ae4b-fa127d31c463/istio-proxy/0.log" Nov 01 23:12:08.338923 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:08.338892 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19969cf-d522-4116-9081-e755269f22d5" path="/var/lib/kubelet/pods/e19969cf-d522-4116-9081-e755269f22d5/volumes" Nov 01 23:12:08.598740 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:08.598714 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rvxmv_96c96ea8-9914-43f7-a349-2a8322a7839c/payload-pre-processing/0.log" Nov 01 23:12:09.426428 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:09.426399 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5758p_20f8556e-bad5-456f-ad1a-14ef9a2ea949/main/0.log" Nov 01 23:12:09.751567 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:09.751442 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-qkfnz_58de7f57-d10e-42ff-ab3e-3952b601bfce/main/0.log" Nov 01 23:12:09.919680 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:09.919652 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-lxpj5_ce3f4a45-aa4a-465f-a436-9c0f4a542380/main/0.log" Nov 01 23:12:10.080699 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:10.080596 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8w866h_ff725c3c-3a7c-41b6-9efb-6ce0ea30ef9e/main/0.log" Nov 01 23:12:10.238356 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:10.238325 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-7klkb_66937625-7220-44c9-b11a-dbc627d0219c/main/0.log" Nov 01 23:12:10.400960 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:10.400931 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-bnbjh_7006c77f-be7e-43be-b47d-7aaef13e5cc6/main/0.log" Nov 01 23:12:19.280170 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:12:19.280132 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:12:24.023301 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:24.023257 6466 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7"] Nov 01 23:12:24.025965 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:24.025936 6466 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-wjbp7"] Nov 01 23:12:24.339325 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:24.339246 6466 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43085e1-d107-4fce-a334-dcd149a4465c" path="/var/lib/kubelet/pods/f43085e1-d107-4fce-a334-dcd149a4465c/volumes" Nov 01 23:12:41.708196 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:41.708163 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b57d74f86-55cvs_c30bab32-ebb2-4bc9-93b4-9cd67b593e4f/manager/0.log" Nov 01 23:12:41.966370 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:12:41.966282 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b57d74f86-55cvs_c30bab32-ebb2-4bc9-93b4-9cd67b593e4f/manager/0.log" Nov 01 23:12:49.278817 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:12:49.278780 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:13:19.281376 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:13:19.281295 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:13:21.268067 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:21.268030 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b57d74f86-55cvs_c30bab32-ebb2-4bc9-93b4-9cd67b593e4f/manager/0.log" Nov 01 23:13:21.514134 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:21.514084 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b57d74f86-55cvs_c30bab32-ebb2-4bc9-93b4-9cd67b593e4f/manager/0.log" Nov 01 23:13:22.286503 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:22.286468 6466 scope.go:122] "RemoveContainer" containerID="dd658257a7aedb6abb64e7d81f246f01d5ea70972b2709f86494cd87812e0b53" Nov 01 23:13:22.296144 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:22.296122 6466 scope.go:122] "RemoveContainer" containerID="ffae4c9aec2c00c5dca5651db54dc45ee42554a45881fd245250d3aa515cab3e" Nov 01 23:13:26.413249 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:26.413209 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cg4rv_8739003e-fba1-4d47-80e9-0f484d117b73/global-pull-secret-syncer/0.log" Nov 01 23:13:26.682630 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:26.682545 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fxlz9_e1baadc8-f14c-490f-9655-a55352e7eb8b/konnectivity-agent/0.log" Nov 01 23:13:26.780157 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:26.780124 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-88.us-east-2.compute.internal_da9c6d0ee7daac240eb44f117e691eaf/haproxy/0.log" Nov 01 23:13:33.059524 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:33.059499 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fqs5h_1f4e4413-8e24-4a65-937d-cc4a9bc0bc25/kuadrant-console-plugin/0.log" Nov 01 23:13:34.669903 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:34.669874 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93531b14-ea0b-44b7-91c6-e99c934595a1/alertmanager/0.log" Nov 01 23:13:34.695573 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:34.695539 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93531b14-ea0b-44b7-91c6-e99c934595a1/config-reloader/0.log" Nov 01 23:13:34.721167 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:34.721136 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93531b14-ea0b-44b7-91c6-e99c934595a1/kube-rbac-proxy-web/0.log" Nov 01 23:13:34.745016 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:34.744992 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93531b14-ea0b-44b7-91c6-e99c934595a1/kube-rbac-proxy/0.log" Nov 01 23:13:34.769117 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:34.769081 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93531b14-ea0b-44b7-91c6-e99c934595a1/kube-rbac-proxy-metric/0.log" Nov 01 23:13:34.795488 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:34.795461 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93531b14-ea0b-44b7-91c6-e99c934595a1/prom-label-proxy/0.log" Nov 01 23:13:34.825171 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:34.825144 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93531b14-ea0b-44b7-91c6-e99c934595a1/init-config-reloader/0.log" Nov 01 23:13:34.926773 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:34.926693 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-zk4kv_e9ed2868-24ad-473c-bac4-01344c0e4bed/kube-state-metrics/0.log" Nov 01 23:13:34.948547 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:34.948522 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-zk4kv_e9ed2868-24ad-473c-bac4-01344c0e4bed/kube-rbac-proxy-main/0.log" Nov 01 23:13:34.973229 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:34.973201 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-zk4kv_e9ed2868-24ad-473c-bac4-01344c0e4bed/kube-rbac-proxy-self/0.log" Nov 01 23:13:35.006756 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.006723 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-796bf69784-66sjq_18595f28-dc19-4df0-a61b-905e22653608/metrics-server/0.log" Nov 01 23:13:35.155034 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.155002 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cw7qc_d77255dd-93d5-4c30-8a4c-f4c7ea44d702/node-exporter/0.log" Nov 01 23:13:35.181136 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.181044 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cw7qc_d77255dd-93d5-4c30-8a4c-f4c7ea44d702/kube-rbac-proxy/0.log" Nov 01 23:13:35.207638 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.207610 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cw7qc_d77255dd-93d5-4c30-8a4c-f4c7ea44d702/init-textfile/0.log" Nov 01 23:13:35.325692 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.325664 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-z8mkz_e8572e79-8cf8-4b15-b77f-ba409bd3b51f/kube-rbac-proxy-main/0.log" Nov 01 23:13:35.348389 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.348365 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-z8mkz_e8572e79-8cf8-4b15-b77f-ba409bd3b51f/kube-rbac-proxy-self/0.log" Nov 01 23:13:35.372755 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.372722 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-z8mkz_e8572e79-8cf8-4b15-b77f-ba409bd3b51f/openshift-state-metrics/0.log" Nov 01 23:13:35.710501 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.710470 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bdf8c4fd-s4wbr_c034008e-a79f-4ebb-9316-d5d63bc1dbb9/telemeter-client/0.log" Nov 01 23:13:35.747250 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.747224 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bdf8c4fd-s4wbr_c034008e-a79f-4ebb-9316-d5d63bc1dbb9/reload/0.log" Nov 01 23:13:35.770756 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.770734 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bdf8c4fd-s4wbr_c034008e-a79f-4ebb-9316-d5d63bc1dbb9/kube-rbac-proxy/0.log" Nov 01 23:13:35.806147 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.806083 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-675964fd8b-v45n9_7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/thanos-query/0.log" Nov 01 23:13:35.828296 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.828268 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-675964fd8b-v45n9_7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/kube-rbac-proxy-web/0.log" Nov 01 23:13:35.850948 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.850917 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-675964fd8b-v45n9_7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/kube-rbac-proxy/0.log" Nov 01 23:13:35.889342 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.889320 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-675964fd8b-v45n9_7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/prom-label-proxy/0.log" Nov 01 23:13:35.912477 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.912450 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-675964fd8b-v45n9_7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/kube-rbac-proxy-rules/0.log" Nov 01 23:13:35.936374 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:35.936350 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-675964fd8b-v45n9_7cd5ecdb-5bcd-44ea-8b20-07bdfe8e2f55/kube-rbac-proxy-metrics/0.log" Nov 01 23:13:36.350630 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.350598 6466 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw"] Nov 01 23:13:36.351041 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.351028 6466 cpu_manager.go:414] "containerMap: removing container" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" Nov 01 23:13:36.351121 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.351044 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" Nov 01 23:13:36.351121 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.351060 6466 cpu_manager.go:414] "containerMap: removing container" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" Nov 01 23:13:36.351121 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.351068 6466 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" Nov 01 23:13:36.351294 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.351185 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" Nov 01 23:13:36.351294 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.351200 6466 memory_manager.go:365] "RemoveStaleState removing state" podUID="48108ced-26bd-4bfd-832e-b14e3f36c1f8" containerName="istio-proxy" Nov 01 23:13:36.355961 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.355942 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.362478 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.362450 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw"] Nov 01 23:13:36.457860 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.457827 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w64nw\"/\"kube-root-ca.crt\"" Nov 01 23:13:36.457860 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.457838 6466 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w64nw\"/\"openshift-service-ca.crt\"" Nov 01 23:13:36.457860 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.457839 6466 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w64nw\"/\"default-dockercfg-nlr9s\"" Nov 01 23:13:36.515211 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.515174 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-lib-modules\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.515211 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.515211 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-proc\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.515424 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.515346 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-sys\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.515424 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.515393 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-podres\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.515493 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.515471 6466 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscz4\" (UniqueName: \"kubernetes.io/projected/7c84f86d-ee1c-4302-b492-d97089560605-kube-api-access-nscz4\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.616879 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.616779 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-sys\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.616879 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.616862 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-podres\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.617142 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.616907 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-sys\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.617142 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.616914 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscz4\" (UniqueName: \"kubernetes.io/projected/7c84f86d-ee1c-4302-b492-d97089560605-kube-api-access-nscz4\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.617142 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.616979 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-podres\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.617142 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.616991 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-lib-modules\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.617142 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.617012 6466 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-proc\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.617142 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.617131 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-lib-modules\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.617142 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.617142 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c84f86d-ee1c-4302-b492-d97089560605-proc\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.626056 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.626033 6466 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscz4\" (UniqueName: \"kubernetes.io/projected/7c84f86d-ee1c-4302-b492-d97089560605-kube-api-access-nscz4\") pod \"perf-node-gather-daemonset-nl8sw\" (UID: \"7c84f86d-ee1c-4302-b492-d97089560605\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.669878 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.669841 6466 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:36.943019 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:36.942986 6466 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw"] Nov 01 23:13:36.946205 ip-10-0-134-88 kubenswrapper[6466]: W1101 23:13:36.946179 6466 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7c84f86d_ee1c_4302_b492_d97089560605.slice/crio-baac99271c55407a8b9accc4270d3e635b49e651cdee7067fa2f1b7b7094ea6f WatchSource:0}: Error finding container baac99271c55407a8b9accc4270d3e635b49e651cdee7067fa2f1b7b7094ea6f: Status 404 returned error can't find the container with id baac99271c55407a8b9accc4270d3e635b49e651cdee7067fa2f1b7b7094ea6f Nov 01 23:13:37.074521 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:37.074488 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" event={"ID":"7c84f86d-ee1c-4302-b492-d97089560605","Type":"ContainerStarted","Data":"ff05fcf966fdb38d026aa2791dd5a4d50c38dae5d90cb7c01c230719fa0a4ea9"} Nov 01 23:13:37.074670 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:37.074540 6466 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" event={"ID":"7c84f86d-ee1c-4302-b492-d97089560605","Type":"ContainerStarted","Data":"baac99271c55407a8b9accc4270d3e635b49e651cdee7067fa2f1b7b7094ea6f"} Nov 01 23:13:37.074670 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:37.074629 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:37.090964 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:37.090919 6466 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" podStartSLOduration=1.09090722 podStartE2EDuration="1.09090722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 23:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:37.090147733 +0000 UTC m=+3020.874586655" watchObservedRunningTime="2026-11-01 23:13:37.09090722 +0000 UTC m=+3020.875346168" Nov 01 23:13:40.206625 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:40.206597 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sh7c7_26366d8c-76f1-479e-9715-6512f20bd3c6/dns/0.log" Nov 01 23:13:40.270886 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:40.270852 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sh7c7_26366d8c-76f1-479e-9715-6512f20bd3c6/kube-rbac-proxy/0.log" Nov 01 23:13:40.421644 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:40.421620 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mfnxk_76a5a648-0585-408d-8556-c0e31dcea942/dns-node-resolver/0.log" Nov 01 23:13:41.117586 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:41.117545 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f9snb_6962a1b2-43de-4da8-a1f4-af8101d42fd0/node-ca/0.log" Nov 01 23:13:43.091156 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:43.091100 6466 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-nl8sw" Nov 01 23:13:44.910090 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:44.910057 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vpkhv_6312a0f9-4dea-44d8-94a5-35f1ee409f10/discovery/0.log" Nov 01 23:13:45.307961 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:45.307881 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d9489c85-8f7t9_3cea7bcd-f62d-4528-b266-4d3a022c7de3/kube-auth-proxy/0.log" Nov 01 23:13:45.376982 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:45.376954 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-44l6s_5a202d72-2f71-4a71-aba6-df821c076507/istio-proxy/0.log" Nov 01 23:13:45.667879 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:45.667835 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-wfrzw_06103ca9-e37d-4286-960e-cd1c06da76af/istio-proxy/0.log" Nov 01 23:13:45.668090 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:45.668048 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-wfrzw_06103ca9-e37d-4286-960e-cd1c06da76af/istio-proxy/1.log" Nov 01 23:13:45.695277 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:45.695251 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-wg8g5_65c9f274-eb9e-4f06-ae4b-fa127d31c463/istio-proxy/0.log" Nov 01 23:13:45.759570 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:45.759542 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rvxmv_96c96ea8-9914-43f7-a349-2a8322a7839c/payload-pre-processing/0.log" Nov 01 23:13:48.018886 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:48.018857 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-77bdk_73bfd9f9-84c5-48b4-bbd1-d1f0611a8b2d/serve-healthcheck-canary/0.log" Nov 01 23:13:48.709827 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:48.709792 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kc74w_ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0/kube-rbac-proxy/0.log" Nov 01 23:13:48.736586 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:48.736549 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kc74w_ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0/exporter/0.log" Nov 01 23:13:48.769166 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:48.769131 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kc74w_ccc2d08c-c26b-41b1-a36d-76dc6ab9fcf0/extractor/0.log" Nov 01 23:13:49.286587 ip-10-0-134-88 kubenswrapper[6466]: E1101 23:13:49.286547 6466 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kc74w" containerName="extractor" Nov 01 23:13:49.640091 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:49.640055 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vpkhv_6312a0f9-4dea-44d8-94a5-35f1ee409f10/discovery/0.log" Nov 01 23:13:49.652506 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:49.652479 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d9489c85-8f7t9_3cea7bcd-f62d-4528-b266-4d3a022c7de3/kube-auth-proxy/0.log" Nov 01 23:13:49.690710 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:49.690678 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-44l6s_5a202d72-2f71-4a71-aba6-df821c076507/istio-proxy/0.log" Nov 01 23:13:49.846830 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:49.846796 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-wfrzw_06103ca9-e37d-4286-960e-cd1c06da76af/istio-proxy/0.log" Nov 01 23:13:49.847024 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:49.846921 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-wfrzw_06103ca9-e37d-4286-960e-cd1c06da76af/istio-proxy/1.log" Nov 01 23:13:49.863570 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:49.863546 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-wg8g5_65c9f274-eb9e-4f06-ae4b-fa127d31c463/istio-proxy/0.log" Nov 01 23:13:49.894909 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:49.894831 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rvxmv_96c96ea8-9914-43f7-a349-2a8322a7839c/payload-pre-processing/0.log" Nov 01 23:13:51.366611 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:51.366579 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8487c6b6d6-mvdmd_cf859056-1280-4da9-a120-1144db7ae3c7/manager/0.log" Nov 01 23:13:51.384411 ip-10-0-134-88 kubenswrapper[6466]: I1101 23:13:51.384387 6466 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8487c6b6d6-mvdmd_cf859056-1280-4da9-a120-1144db7ae3c7/copy-manifests/0.log"